Method and apparatus for enhancing three-dimensional imagery data

• US 8,958,654 B1
• Filed: 03/08/2004
• Issued: 02/17/2015
• Est. Priority Date: 04/25/2001
• Status: Active Grant

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Evans, Bruno J., Tener, Gene D.
• Primary Examiner(s): Thirugnanam, Gandhi
• Application Number:
  US10/795,779
• Time in Patent Office: 3,998 Days
• Field of Search: 382/103, 382/106, 382/151, 382/154, 382/209, 382/274, 382/275, 382/294
• US Class Current: 382/254
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A method and apparatus for enhancing the resolution of three-dimensional imagery data are disclosed. The method includes registering a frame of the three-dimensional imagery data with a template frame; and temporally averaging the registered frame with the template frame. The apparatus includes a program storage medium encoded with instructions that, when executed by a computing apparatus, performs the method and an apparatus programmed to perform the method.

Claims

1. A computer-implemented method, comprising:

accessing a set of three-dimensional imagery data; and

processing the three-dimensional imagery data to enhance its spatial resolution including;

registering a first frame of the three-dimensional imagery data with a first template frame to create a registered first frame;

temporally filtering the registered first frame with the first template frame;

registering a second frame of two-dimensional imagery data corresponding to the first frame of the three-dimensional imagery data with a second template frame of the two-dimensional imagery data corresponding to the first template frame to create a registered second frame; and

temporally filtering the registered second frame of the two-dimensional imagery data with the second template frame of the two-dimensional imagery data;

wherein the accessing and the processing are performed by a computing apparatus.

2. The computer-implemented method of claim 1, wherein registering the first frame of the three-dimensional imagery data includes:

correlating the first frame of the three-dimensional imagery data to the first template frame to create a correlated frame; and

performing a three-dimensional interpolation on the correlated frame.

3. The computer-implemented method of claim 2, further comprising sampling the correlated frame.

4. The computer-implemented method of claim 1, wherein processing the three-dimensional imagery data further includes at least one of:

spatially filtering the first frame of the three-dimensional imagery data prior to registration;

updating the first template frame;

replacing the first template frame; and

spatially filtering the temporally filtered frame created by temporally filtering the registered first frame with the first template frame.

5. A non-transitory program storage medium encoded with instructions that, when executed by a computing apparatus, perform a method comprising:

accessing a set of three-dimensional imagery data; and

processing the three-dimensional imagery data to enhance its spatial resolution including;

registering a first frame of the three-dimensional imagery data with a first template frame to create a registered first frame;

temporally filtering the registered first frame with the first template frame;

registering a second frame of two-dimensional imagery data corresponding to the first frame of the three-dimensional imagery data with a second template frame of the two-dimensional imagery data corresponding to the first template frame to create a registered second frame; and

temporally filtering the registered second frame of the two-dimensional imagery data with the second template frame of the two-dimensional imagery data;

wherein the accessing and the processing are performed by the computing apparatus.

6. The non-transitory program storage medium of claim 5, wherein registering the first frame of the three-dimensional imagery data in the encoded method includes:

correlating the first frame of the three-dimensional imagery data to the first template frame to create a correlated frame; and

performing a three-dimensional interpolation on the correlated frame.

7. The non-transitory program storage medium of claim 6, wherein processing the three-dimensional imagery data in the encoded method further includes sampling the correlated frame.

8. The non-transitory program storage medium of claim 5, wherein processing the three-dimensional imagery data in the encoded method further includes at least one of:

spatially filtering the first frame of the three-dimensional imagery data prior to registration;

updating the first template frame;

replacing the first template frame; and

spatially filtering the temporally filtered frame created by temporally filtering the registered first frame with the first template frame.

9. An apparatus, comprising:

a processor;

an electronic storage from which the processor may read;

a set of three-dimensional imagery data residing in the electronic storage; and

an application residing in the electronic storage and performing, when executed by the processor, a method comprising;

accessing the set of three-dimensional imagery data; and

processing the three-dimensional imagery data to enhance its spatial resolution including;

registering a first frame of the three-dimensional imagery data with a first template frame to create a registered first frame;

temporally filtering the registered first frame with the first template frame;

registering a second frame of two-dimensional imagery data corresponding to the first frame of the three-dimensional imagery data with a second template frame of the two-dimensional imagery data corresponding to the first template frame to create a registered second frame; and

temporally filtering the registered second frame of the two-dimensional imagery data with the second template frame of the two-dimensional imagery data.

10. The apparatus of claim 9, wherein registering the first frame of the three-dimensional imagery data in the method includes:

correlating the first frame of the three-dimensional imagery data to the first template frame to create a correlated frame; and

performing a three-dimensional interpolation on the correlated frame.

11. The apparatus of claim 10, wherein processing the three-dimensional imagery data in the method further includes sampling the correlated frame.

12. The apparatus of claim 9, wherein processing the three-dimensional imagery data in the method further includes at least one of:

spatially filtering the first frame of the set of three-dimensional imagery data prior to registration;

updating the first template frame;

replacing the first template frame; and

spatially filtering the temporally filtered frame created by temporally filtering the registered first frame with the first template frame.

13. An apparatus, comprising:

a data acquisition subsystem, comprising a processor, capable of acquiring a set of three-dimensional imagery data; and

a data processing subsystem, comprising the processor, programmed to;

access the set of three-dimensional imagery data; and

process the three-dimensional imagery data to enhance its spatial resolution including to;

register a first frame of the three-dimensional imagery data with a first template frame to create a registered first frame;

temporally filter the registered first frame with the first template frame;

register a second frame of two-dimensional imagery data corresponding to the first frame of the three-dimensional imagery data with a second template frame of the two-dimensional imagery data corresponding to the first template frame to create a registered second frame; and

temporally filter the registered second frame of the two-dimensional imagery data with the second template frame of the two-dimensional imagery data.

14. The apparatus of claim 13, wherein the data acquisition subsystem includes:

a laser capable of emitting optical radiation; and

a detector subsystem capable of receiving the emitted optical radiation upon its reflection.

15. The apparatus of claim 13, wherein three-dimensional imagery data is LADAR data or stereo imaging data.

16. The apparatus of claim 13, wherein registering the first frame of the three-dimensional imagery data includes:

correlating the first frame of the three-dimensional imagery data to the first template frame to create a correlated frame; and

performing a three-dimensional interpolation on the correlated frame.

17. The apparatus of claim 13, wherein the data processing subsystem is further capable of tracking an object in a field of view for the data acquisition subsystem.

18. The apparatus of claim 13, wherein the data processing subsystem is further capable of:

converting the first frame of the three-dimensional imagery data to Cartesian coordinates to create a converted frame; and

capturing the converted frame.

19. The apparatus of claim 13, wherein the data processing subsystem is further capable of:

spatially filtering the first frame of the three-dimensional imagery data prior to registration;

updating the first template frame;

replacing the first template frame; and

spatially filtering the temporally filtered frame created by temporally filtering the registered first frame with the first template frame.

20. The apparatus of claim 13, wherein the apparatus comprises a portion of an airborne platform.

21. The apparatus of claim 13, wherein the data acquisition subsystem comprises a portion of a platform and the data processing subsystem is located remotely from the platform.

22. The apparatus of claim 13, further comprising a display.

23. A computer-implemented method, comprising:

accessing a set of three-dimensional imagery data; and

processing the three-dimensional imagery data to enhance its spatial resolution including;

registering a frame of the three-dimensional imagery data with a template frame to create a registered first frame including;

correlating an intensity image of a new frame of LADAR data to an intensity image of the template frame; and

correlating a three-dimensional image of the new frame to a three-dimensional image of the template frame;

temporally filtering the registered first frame with the template frame;

wherein the accessing and the processing are performed by a computing apparatus.

24. A computer-implemented method for enhancing LADAR data resolution, comprising:

correlating an intensity image of a new frame of laser detection and ranging (LADAR) data to an intensity image of a template frame;

correlating a three-dimensional image of the new frame to a three-dimensional image of the template frame to create a correlated three-dimensional image;

performing a sub-voxel interpolation on the correlated three-dimensional image;

registering the new frame with the template frame subsequent to correlation and interpolation; and

temporally filtering the registered new frame with the template frame;

wherein the correlating, the interpolating, the registration, and the temporal filtering are performed by a processor.

25. The computer-implemented method of claim 24, wherein correlating the intensity image of the new frame to the intensity image of the template frame includes:

subtracting the intensity image of the template frame from the intensity image of the new frame in a point wise fashion; and

outputting a shift vector.

26. The computer-implemented method of claim 24, wherein registering the new frame with the template frame includes performing a sub-pixel interpolation on a predetermined region around a center pixel of a target in the intensity image of the new frame to locate a true centroid.

27. The computer-implemented method of claim 26, wherein the predetermined region is a 5×

5 pixel region.

28. The computer-implemented method of claim 24, further comprising re-sampling at least one of the new frame and the template frame.

29. The computer-implemented method of claim 24, further comprising acquiring three-dimensional imagery data.

30. The computer-implemented method of claim 29, wherein acquiring the three-dimensional imagery data includes:

detecting a signal whose characteristics are represented by the three-dimensional imagery data;

capturing the three-dimensional imagery data; and

storing the three-dimensional imagery data.

31. The computer-implemented method of claim 24, further comprising further enhancing the resolution of the temporally filtered frame for display.

32. The computer-implemented method of claim 24, further comprising displaying the temporally filtered frame.