Fusion of a 2D electro-optical image and 3D point cloud data for scene interpretation and registration performance assessment

• US 8,179,393 B2
• Filed: 02/13/2009
• Issued: 05/15/2012
• Est. Priority Date: 02/13/2009
• Status: Active Grant

• Current Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Original Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Inventors: Minear, Kathleen, Pooley, Donald, Smith, Anthony O'Neil
• Primary Examiner(s): Nguyen, Phu K
• Application Number:
  US12/371,186
• Publication Number:
  US 20100207936A1
• Time in Patent Office: 1,187 Days
• Field of Search: 345/419, 345/422, 345/619, 345/642, 382/154, 382/190, 382/291, 382/294, 715/851
• US Class Current: 345/419
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration

Abstract

Method and system for combining a 2D image with a 3D point cloud for improved visualization of a common scene as well as interpretation of the success of the registration process. The resulting fused data contains the combined information from the original 3D point cloud and the information from the 2D image. The original 3D point cloud data is color coded in accordance with a color map tagging process. By fusing data from different sensors, the resulting scene has several useful attributes relating to battle space awareness, target identification, change detection within a rendered scene, and determination of registration success.

Claims

1. A method performed by a computing device for combining a 2D image with a 3D image for improved visualization of a common scene, comprising:

analyzing said 2D image to identify selected content based characteristics of a plurality of first areas in said common scene;

selectively assigning to each area of said plurality of first areas a different color map tag of a plurality of color map tags that corresponds to a content based characteristic of said area, each of said plurality of color map tags identifying a different one of a plurality of color maps;

using said plurality of color map tags to assign a different color map of said plurality of color maps to each of a plurality of second areas of said 3D image which correspond to said plurality of first areas of said 2D image;

respectively applying said plurality of color maps to data defining said plurality of second areas of said 3D image;

forming a virtual 3D image from said 2D image by assigning a Z value to pixels in said 2D image based on a contour of a surface defined by first points of said 3D image;

determining color values for second points in the virtual 3D image based on a desired color map; and

creating a fused image by overlaying said 3D image and said virtual 3D image.

2. The method according to claim 1, further comprising evaluating a performance or quality of a registration process by visually inspecting said fused image to determine if features in said common scene are properly aligned.

3. The method according to claim 1, wherein said content based characteristics are selected from the group consisting of urban content, natural content, water content, and man-made structure content.

4. The method according to claim 3, wherein said man-made structure content is selected from the group consisting of buildings, houses, roadways, and vehicles.

5. The method according to claim 1, wherein said 3D image is comprised of a plurality of points comprising a 3D point cloud where each point is defined in accordance with an X, Y, and Z coordinate axis value, and said 2D image is comprised of a plurality of pixels having a position defined exclusively in accordance with said X and Y coordinate axis values.

6. The method according to claim 5, further comprising assigning each of said plurality of color maps to one or more points having X, Y, and Z coordinate values within areas in said 3D image based on said plurality of color map tags assigned to corresponding ones of said X, Y coordinate values of said plurality of areas identified in said 2D image.

7. The method according to claim 1, further comprising removing a portion of said 3D image comprising data defining said surface prior to creating said fused image.

8. The method according to claim 5, wherein said values are assigned by:

assigning to at least a first one of said plurality of pixels of said 2D image a first Z value of a corresponding one of said first points of said 3D image that has an X coordinate value and a Y coordinate value that is the same thereof; and

interpolating or estimating a second Z value for at least a second one of said plurality of pixels of said 2D image if none of said first points in said 3D image has the same X coordinate value and Y coordinate value as said one of said plurality of pixels.

9. The method according to claim 1, wherein at least one color map of said plurality of color maps is selected to mimic colors or hues that are commonly associated with said content based characteristic of the area for which the color map is used.

10. The method according to claim 1, further comprising registering said 2D image and said 3D image.

11. A system for combining a 2D image with a 3D image for improved visualization of a common scene, comprising:

a computer programmed with a set of instructions for

analyzing said 2D image to identify selected content based characteristics of a plurality of first areas in said common scene;

selectively assigning to each area of said plurality of first areas a different color map tag of a plurality of color map tags that corresponds to a content based characteristic of said area, each of said plurality of color map tags identifying a different one of a plurality of color maps;

using said plurality of color map tags to assign a different color map of said plurality of color maps to each of a plurality of second areas of said 3D image which correspond to said plurality of first areas of said 2D image;

respectively applying said plurality of color maps to data defining said plurality of second areas of said 3D image;

forming a virtual 3D image from said 2D image by assigning a Z value to pixels in said 2D image, each said Z value determined based on a contour of a surface defined by first points of said 3D image;

determining color values for second point in the virtual 3D image based on a desired color map on a color values of a corresponding pixel in said 2D image; and

creating a fused image by overlaying said 3D image and said virtual 3D image.

12. The system according to claim 11, further comprising evaluating a performance or quality of a registration process by visually inspecting said fused image to determine if features in said common scene are properly aligned.

13. The system according to claim 11, wherein said content based characteristics are selected from the group consisting of urban content, natural content, water content, and man-made structure content.

14. The system according to claim 13, wherein said man-made structure content is selected from the group consisting of buildings, houses, roadways, and vehicles.

15. The system according to claim 11, wherein said 3D image is comprised of a plurality of points comprising a 3D point cloud where each point is defined in accordance with an X, Y, and Z coordinate axis value, and said 2D image is comprised of a plurality of pixels having a position defined exclusively in accordance with X and Y coordinate axis values.

16. The system according to claim 15, wherein said computer is programmed to assign each of said plurality of color maps to one or more points having X, Y, and Z coordinate values within areas in said 3D image based on said plurality of color map tags assigned to corresponding ones of said X, Y coordinate values of said plurality of areas identified in said 2D image.

17. The system according to claim 11, wherein said computer is programmed to remove a portion of said 3D image comprising data defining said surface prior to creating said fused image.

18. The system according to claim 15, wherein said Z values are assigned by:

assigning to at least a first one of said plurality of pixels of said 2D image a first Z value of a corresponding one of said first points of said 3D image that has an X coordinate value and a Y coordinate value that is the same thereof; and

interpolating or estimating a second Z value for at least a second one of said plurality of pixels of said 2D image if none of said first data points in said 3D image has the same X coordinate value and Y coordinate value as said one of said plurality of 2D pixels.

19. The system according to claim 11, wherein at least one color map of said plurality of color maps mimics colors or hues that are commonly associated with said content based characteristic of the area for which the color map is used.

20. The system according to claim 11, wherein said computer is programmed to register said 2D image and said 3D image.

21. A method performed by a computing device for combining a 2D image with a 3D image for improved visualization of a common scene, comprising:

analyzing said 2D image to identify selected content based characteristics of a plurality of first areas in said common scene;

selectively assigning to each area of said plurality of first areas a different color map tag of a plurality of color map tags that correspond to a content based characteristic of said area, each of said plurality of color map tags identifying a different one of a plurality of color maps;

registering a 2D image with a 3D image;

using said plurality of color map tags to assign a different color map of said plurality of color maps to each of a plurality of second areas of said 3D image which correspond to said plurality of first areas of said 2D image;

respectively applying said plurality of color maps to data defining said plurality of second areas of said 3D image;

forming a virtual 3D image from said 2D image by assigning a Z value to pixels in said 2D image based on a contour of a surface defined by first points of said 3D image; and

creating a fused image by overlaying said 3D image and said virtual 3D image in accordance with said registration.