Method and system for providing a perspective view image by intelligent fusion of a plurality of sensor data

• US 20080158256A1
• Filed: 06/25/2007
• Published: 07/03/2008
• Est. Priority Date: 06/26/2006
• Status: Abandoned Application

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Russell, Richard, Hoehn, Terence, Shepherd, Alexander T.
• Application Number:
  US11/819,149
• Publication Number:
  US 20080158256A1
• Time in Patent Office: Days
• US Class Current: 345/629
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2380/10 Automotive applications
• CPC Class Codes: G09G 2380/12 Avionics applications

Abstract

A method and system for providing a perspective view image created by fusing a plurality of sensor data for supply to a platform operator with a desired viewing perspective within an area of operation is disclosed. A plurality of sensors provide substantially real-time data of an area of operation, a processor combines the substantially real-time data of the area of operation with digital terrain elevation data of the area of operation and positional data of a platform operator to create a digital cartographic map database having substantially real-real time sensor data, a memory for storing the digital cartographic map database, a perspective view data unit inputs data regarding a desired viewing perspective of the operator within the area of operation with respect to the digital cartographic map database to provide a perspective view image of the area of operation, and a display for displaying the perspective view image to the operator.

Claims

1. A method for providing a perspective view image created by fusing a plurality of sensor data for supply to an operator with a desired viewing perspective within an area of operation, wherein the area of operation includes a battlefield, comprising:

providing a plurality of sensors configured to provide substantially real-time data of the area of operation;

combining the substantially real-time data of the area of operation with digital terrain elevation data of the area of operation and positional data of the operator to create a digital cartographic map database having substantially real-real time sensor data;

inputting data regarding the desired viewing perspective within the area of operation with respect to the digital cartographic map database to provide a perspective view image of the area of operation; and

displaying the perspective view image to the operator.

2. The method of claim 1, further comprising:

receiving updated positional data regarding the operator'"'"'s current position; and

updating the cartographic map database to reflect the operator'"'"'s current position based on the updated positional data.

3. The method of claim 1, further comprising:

receiving updated perspective view data through six-degree-of-freedom steering inputs from the operator, either from manual or head-steered commands; and

updating the displayed perspective view image in accordance with the received updated perspective view data.

4. The method of claim 1, wherein the plurality sensors includes one or more of the following image sensors:

electro-optical (EO) image sensor, infrared (IR) image sensor, intensified or low-light level image sensor, radar three dimensional image sensor, or range data image sensor, or human eye.

5. The method of claim 4, wherein sensor data includes compressed still or motion imagery.

6. The method of claim 4, wherein sensor data includes raw still or motion imagery.

7. The method of claim 1, further comprising displaying the perspective view image on one of the following display devices:

Cathode Ray Tubes (CRT), flat-panel solid state display, helmet mounted devices (HMD), and optical projection heads-up displays (HUD).

8. The method of claim 1, further comprising:

creating a remote Tactical Situational Awareness Registry (TSAR) for storing situational awareness data obtained through six-degree-of-freedom location awareness inputs;

and providing the situational awareness data to the operator that is not contained or available locally by the operator.

9. The method of claim 8, further comprising:

providing a communication path to the operator to acquire the situational awareness data requested by the operator based on a profile of the operator.

10. The method of claim 1, further comprising:

creating a three-dimensional digital cartographic map database of the area of operation.

11. The method of claim 1, further comprising:

receiving a plurality of imagery through an application interface, wherein the imagery includes still and motion imagery in multiple color bands or wavelengths; and

designating a set of metadata corresponding to the plurality of imagery for providing a path into a visual application including the digital cartographic map database.

12. The method of claim 11, further comprising:

synchronizing the set of metadata with the plurality of imagery.

13. The method of claim 1, further comprising:

utilizing the digital cartographic map database to provide a framework for scalable and various degrees of multi-sensor fusion with two-dimensional and three-dimensional RF and EO imaging sensors and other intelligence sources.

14. The method of claim 13, further comprising:

adding geo-location data to individual video frames to allow referencing each sensor data with respect to the other imaging sensors and to the digital cartographic map database.

15. The method of claim 1, further comprising:

utilizing two-dimensional and three-dimensional RF, imaging and other sensor data as truth source for difference detection against the digital cartographic map database.

16. The method of claim 1, further comprising:

seamlessly translating the digital cartographic map data stored on the digital cartographic map database from mission planning/rehearsal simulation into tactical real-time platform and mission environment.

17. A system for providing a perspective view image created by fusing a plurality of sensor data for supply to an operator with a desired viewing perspective within an area of operation, wherein the area of operation includes a battlefield, comprising:

a receiver for receiving a plurality of substantially real-time sensor data of the area of operation from a plurality of sensors;

a processor for combining the substantially real-time sensor data of the area of operation with digital terrain elevation data of the area of operation and positional data of the operator to create a digital cartographic map database having substantially real-real time sensor data;

a perspective view data unit for inputting data regarding the desired viewing perspective within the area of operation with respect to the digital cartographic map database to provide a perspective view image of the area of operation; and

a display for displaying the perspective view image to the operator.

18. The system of claim 17, wherein the receiver receives updated positional data regarding the operator'"'"'s current position in order to update the cartographic map database to reflect the operator'"'"'s current position based on the updated positional data.

19. The system of claim 17, wherein the receiver receives updated perspective view data from the operator through six-degree-of-freedom steering inputs either from manual or head-steered commands;

in order to update the displayed perspective view image in accordance with the received updated perspective view data.

20. The system of claim 17, wherein the plurality sensors includes one or more of the following image sensors:

electro-optical (EO) image sensor, infrared (IR) image sensor, intensified or low-light level image sensor, radar three dimensional image sensor, or range data image sensor or human eye.

21. The system of claim 20, wherein the sensor data includes compressed still or motion imagery.

22. The system of claim 20, wherein the sensor data includes raw still or motion imagery.

23. The system of claim 17, wherein the display includes one or more of the following devices:

Cathode Ray Tubes (CRT), flat-panel solid state display, helmet mounted devices (HMD), and optical projection heads-up displays (HUD).

24. The system of claim 17, further comprising:

a registry for storing remote tactical situational awareness data obtained through six-degree-of-freedom location awareness inputs wherein the display displays the situational awareness data to the operator that is not contained or available locally by the operator.

25. The system of claim 17, wherein the digital cartographic map database includes three-dimensional digital cartographic map data of the area of operation.

26. The system of claim 17, further comprising:

an application interface for receiving a plurality of imagery, wherein the imagery includes still and motion imagery in multiple color bands or wavelengths; and

a set of metadata corresponding to the plurality of imagery for providing a path into a visual application including the digital cartographic map database.

27. The system of claim 26, wherein the set of metadata is synchronized with the plurality of imagery.

28. The system of claim 17, wherein the digital cartographic map database is utilized to provide a framework for scalable and various degrees of multi-sensor fusion with two-dimensional and three-dimensional RF and EO imaging sensors and other intelligence sources.

29. The system of claim 28, wherein geo-location data is added to individual video frames to allow referencing each sensor data with respect to the other imaging sensors and to the digital cartographic map database.

30. The system of claim 17, further comprising:

utilizing two-dimensional and three-dimensional RF, imaging and other sensor data as truth source for difference detection against the digital cartographic map database.

31. The system of claim 17, wherein the digital cartographic map data stored on the digital cartographic map database is seamlessly translated from mission planning/rehearsal simulation into tactical substantially real-time platform and mission environment.

32. A computer readable storage medium having stored thereon computer executable program for providing a perspective view image created by fusing a plurality of sensor data for supply to an operator with a desired viewing perspective within an area of operation, wherein the area of operation includes a battlefield, the computer program when executed causes a processor to execute steps of:

providing a plurality of sensors configured to provide substantially real-time data of the area of operation;

combining the substantially real-time data of the area of operation with digital terrain elevation data of the area of operation and positional data of the operator to create a digital cartographic map database having substantially real-real time sensor data;

inputting data regarding the desired viewing perspective within the area of operation with respect to the digital cartographic map database to provide a perspective view image of the area of operation; and

displaying the perspective view image to the operator.

33. The computer readable storage medium of claim 32, wherein the computer program when executed causes the processor to further execute steps of:

receiving updated positional data regarding the operator'"'"'s current position; and

updating the cartographic map database to reflect the operator'"'"'s current position based on the updated positional data.

34. The computer readable storage medium of claim 32, wherein the computer program when executed causes the processor to further execute steps of:

receiving updated perspective view data from the operator through six-degree-of-freedom steering inputs; and

updating the displayed perspective view image in accordance with the received updated perspective view data.

35. The computer readable storage medium of claim 32, wherein the plurality sensors includes one or more of the following image sensors:

electro-optical (EO) image sensor, infrared (IR) image sensor, intensified or low-light level image sensor, radar three dimensional image sensor, or range data image sensor.

36. The computer readable storage medium of claim 35, wherein the sensor data includes compressed still or motion imagery.

37. The computer readable storage medium of claim 35, wherein the sensor data includes raw still or motion imagery.

38. The computer readable storage medium of claim 32, wherein the computer program when executed causes the processor to further execute step of displaying the perspective view image on one of the following display devices:

Cathode Ray Tubes (CRT), flat-panel solid state display, helmet mounted devices (HMD), and optical projection heads-up displays (HUD).

39. The computer readable storage medium of claim 32, wherein the computer program when executed causes the processor to further execute steps of:

creating a remote Tactical Situational Awareness Registry (TSAR) for storing situational awareness data obtained through six-degree-of-freedom location awareness inputs;

and providing the situational awareness data to the operator that is not contained or available locally by the operator.

40. The computer readable storage medium of claim 39, wherein the computer program when executed causes the processor to further execute steps of:

providing a communication path to the operator to acquire the situational awareness data requested by the operator based on a profile of the operator.

41. The computer readable storage medium of claim 32, wherein the computer program when executed causes the processor to further execute step of:

creating a three-dimensional digital cartographic map database of the area of operation.

42. The computer readable storage medium of claim 32, wherein the computer program when executed causes the processor to further execute steps of:

receiving a plurality of imagery through an application interface, wherein the imagery includes still and motion imagery in multiple color bands or wavelengths; and

designating a set of metadata corresponding to the plurality of imagery for providing a path into a visual application including the digital cartographic map database.

43. The computer readable storage medium of claim 42, wherein the computer program when executed causes the processor to further execute step of synchronizing the set of metadata with the plurality of imagery.

44. The computer readable storage medium of claim 32, wherein the computer program when executed causes the processor to further execute step of:

utilizing the digital cartographic map database to provide a framework for scalable and various degrees of multi-sensor fusion with two-dimensional and three-dimensional RF and EO imaging sensors and other intelligence sources.

45. The computer readable storage medium of claim 32, wherein the computer program when executed causes the processor to further execute step of:

adding geo-location data to individual video frames to allow referencing each sensor data with respect to the other imaging sensors and to the digital cartographic map database.

46. The computer readable storage medium of claim 32, wherein the computer program when executed causes the processor to further execute step of:

utilizing two-dimensional and three-dimensional RF, imaging and other sensor data as truth source for difference detection against the digital cartographic map database.

47. The computer readable storage medium of claim 32, wherein the computer program when executed causes the processor to further execute step of:

seamlessly translating the digital cartographic map data stored on the digital cartographic map database from mission planning/rehearsal simulation into tactical real-time platform and mission environment.