TRACKING AND IMAGING DATA FUSION

• US 20100157056A1
• Filed: 05/20/2008
• Published: 06/24/2010
• Est. Priority Date: 05/20/2007
• Status: Abandoned Application

• Current Assignee: Rafael Advanced Defense Systems Limited (Government of Israel)
• Original Assignee: Rafael Advanced Defense Systems Limited (Government of Israel)
• Inventors: Zohar, Ron, Reinitz, David
• Application Number:
  US12/600,839
• Publication Number:
  US 20100157056A1
• Time in Patent Office: Days
• US Class Current: 348/144
• CPC Class Codes: G01S 13/66 Radar-tracking systems; Analogous systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/89 for mapping or imaging
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10044 Radar image
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30212 Military
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

A method of monitoring a target, which includes tracking the target using a tracking modality, thereby obtaining an estimated track of the target, imaging the target using an imaging modality, thereby obtaining an image of the target, associating the estimated track with the image, and displaying at least one datum related to the image along with the estimated track.

Claims

1. A method of monitoring a target, comprising the steps of:

(a) tracking the target using a tracking modality, thereby obtaining an estimated track of the target;

(b) imaging the target using an imaging modality, thereby obtaining an image of the target;

(c) associating said estimated track with said image; and

(d) displaying at least one datum related to said image along with said estimated track.

2. The method of claim 1, wherein said tracking modality includes ground moving target indicator radar.

3. The method of claim 1, wherein said tracking modality includes signals intelligence.

4. The method of claim 1, wherein said tracking modality includes at least one electro-optical sensor.

5. The method of claim 1, wherein said tracking modality includes a plurality of sensors and a data fusion mechanism for analyzing data acquired by said sensors.

6. The method of claim 1, wherein said imaging modality is a combined imaging and tracking modality.

7. The method of claim 6, wherein said combined imaging and tracking modality includes video motion detection.

8. The method of claim 6, further comprising the steps of:

(e) tracking the target using said combined imaging and tracking modality; and

(f) correcting said tracking by said tracking modality according to said tracking by said combined imaging and tracking modality.

9. The method of claim 1, further comprising the step of:

(e) identifying the target, based at least in part on at least a portion of said image.

10. The method of claim 9, wherein said identifying provides said at least one datum.

11. The method of claim 9, further comprising the step of:

(f) correcting said tracking of the target by said tracking modality in accordance with said identifying.

12. The method of claim 9, wherein said identifying is based in part on said estimated track.

13. The method of claim 1, wherein said image is a single frame.

14. The method of claim 1, wherein said image is a plurality of frames.

15. The method of claim 1, wherein said associating includes mapping coordinates used by said tracking modality into coordinates used by said imaging modality.

16. The method of claim 1, wherein said associating includes mapping coordinates used by said imaging modality into coordinates used by said tracking modality.

17. The method of claim 1, wherein said displaying is effected substantially simultaneously with said associating.

18. The method of claim 1, wherein said displaying is effected subsequent to said associating.

19. The method of claim 18, further comprising the step of:

(e) archiving said estimated track and said at least one datum, said displaying being effected subsequent to said archiving.

20. The method of claim 19, wherein said displaying is effected upon request of an operator of a system whereat said estimated track and said at least one datum are archived.

21. The method of claim 1, further comprising the steps of:

(e) archiving said estimated track and said image; and

(f) identifying the target, based at least in part on at least a portion of said image as archived, thereby providing said at least one datum.

22. The method of claim 21, wherein said identifying is based in part on said estimated track as archived.

23. The method of claim 1, wherein said imaging includes pointing said imaging modality at the target in accordance with said estimated track.

24. The method of claim 1, further comprising the step of:

(e) moving said imaging modality, in accordance with said estimated track, to a vantage point that facilitates said imaging.

25. The method of claim 24, wherein said moving is effected by steps including moving a platform whereon said imaging modality is mounted.

26. The method of claim 24, wherein said vantage point is selected at least in part in accordance with said estimated track.

27. The method of claim 24, wherein said vantage point is selected at least in part in accordance with a location of at least one object that partly hides the target.

28. The method of claim 1, wherein said displaying is effected at a location different from where said associating is effected, the method further comprising the step of:

(e) transmitting said estimated track along with said at least one datum that is associated therewith to said location where said displaying is effected.

29. The method of claim 1, wherein said at least one datum includes at least a portion of said image that depicts the target.

30. A system for monitoring a target, comprising:

(a) a tracking subsystem for tracking the target, thereby obtaining an estimated track of the target;

(b) an imaging subsystem, separate from said tracking subsystem, for imaging the target, thereby obtaining an image of the target;

(c) an association mechanism for associating said estimated track with said image; and

(d) a display mechanism for displaying at least one datum related to said image along with said estimated track.

31. The system of claim 30, further comprising:

(e) at least one vehicle on which said tracking subsystem and said imaging subsystem are mounted.

32. The system of claim 31, comprising a plurality of said vehicles, with said tracking subsystem and said imaging subsystem being mounted on respective said vehicles.

33. The system of claim 32, further comprising:

(f) a wireless mechanism for exchanging, between said vehicles, results of said tracking and said imaging.

34. The system of claim 32, wherein said mechanism for associating said estimated track with said image is distributed between said vehicles.

35. The system of claim 31, wherein said at least one vehicle is an aircraft.

36. A method of monitoring a plurality of targets, comprising the steps of:

(a) tracking the targets using a tracking modality, thereby obtaining, for each of at least one target group that includes a respective at least one of the targets, a respective estimated track;

(b) for each said target group;

(i) based at least in part on said respective estimated track, imaging and tracking each said at least one respective target of said each target group using a combined imaging and tracking modality, thereby obtaining a respective image of each said respective target, and

(ii) associating said respective estimated track with said respective at least one image; and

(c) selectively displaying at least one said estimated track along with, for at least one of said image that is associated therewith, at least one datum related to said each at least one image.

37. The method of claim 36, wherein said displaying is effected at a location different from where said associating is effected, the method further comprising the step of:

(d) transmitting said at least one estimated track along with said at least one datum that is associated therewith to said location where said displaying is effected.

38. The method of claim 37, wherein, for each said estimated track that is selectively displayed, for each said image that is associated therewith, said at least one datum includes only a portion of said image.

39. The method of claim 38, wherein, for each said estimated track that is selectively displayed, each said image that is associated therewith includes a plurality of frames and said at least one datum that is related to said each image includes only a portion of each said frame.

40. The method of claim 36, wherein, for each of at least one said image that is associated with said at least one estimated track that is displayed, said at least one datum includes textual information about at least a portion of said each at least one image.

41. A system for monitoring a plurality of targets, comprising:

(a) a tracking subsystem for tracking the targets, thereby obtaining, for each of at least one target group that includes a respective at least one of the targets, a respective estimated track;

(b) a plurality of imaging subsystems for imaging the targets, each imaging subsystem for imaging a respective one of the targets, thereby obtaining a respective image that depicts said one target;

(c) an association mechanism for associating each said image with said respective estimated track of said target group that includes the target that is depicted by said each image; and

(d) a display mechanism for selectively displaying at least one said estimated track along with at least one datum related to at least one of each said image that is associated therewith.

42. The system of claim 41, further comprising:

(e) a plurality of vehicles, said tracking subsystem and said imaging subsystem being mounted on respective said vehicles.

43. The system of claim 42, wherein said vehicles are aircraft.

44. A method of monitoring a target, comprising the steps of:

(a) tracking the target, using a tracking modality; and

(b) in response to a degradation of said tracking of the target;

(i) acquiring an image of a region estimated, based at least in part on said tracking of the target, to include the target, and

(ii) locating the target in said region, based at least in part on at least a portion of said image.

45. The method of claim 44, wherein said tracking modality includes ground moving target indicator radar.

46. The method of claim 44, wherein said tracking modality includes signals intelligence.

47. The method of claim 44, wherein said tracking modality includes at least one electro-optical sensor.

48. The method of claim 44, wherein said tracking modality includes a plurality of sensors and a data fusion mechanism for analyzing data acquired by said sensors.

49. The method of claim 44, further comprising the step of:

(c) based at least in part on said locating the target in said region, resuming said tracking of the target.

50. The method of claim 49, wherein said resuming of said tracking is effected using said tracking modality.

51. The method of claim 49, wherein said acquiring of said image is effected using a combined imaging and tracking modality and wherein said resuming of said tracking is effected using said combined imaging and tracking modality.

52. The method of claim 51, wherein said combined imaging and tracking modality includes video motion detection.

53. The method of claim 44, wherein said acquiring includes pointing an imaging modality, that effects said acquiring, at said region.

54. The method of claim 44, further comprising the step of:

(c) in response to said degradation of said tracking of said target, moving an imaging modality to a vantage point that facilitates said acquiring, said acquiring then being effected using said imaging modality.

55. The method of claim 54, wherein said moving is effected by steps including moving a platform whereon said imaging modality is mounted.

56. The method of claim 54, wherein said tracking estimates an estimated track of the target, and wherein said vantage point is selected at least in part in accordance with said estimated track.

57. The method of claim 54, wherein said vantage point is selected at least in part in accordance with a location of at least one object that partly hides the target.

58. The method of claim 44, wherein said locating is based at least in part on a thermal contrast between the target and said region.

59. The method of claim 44, wherein said acquiring and said locating is effected using a combined imaging and tracking modality.

60. The method of claim 59, wherein said combined imaging and tracking modality includes video motion detection.

61. The method of claim 44, further comprising the step of:

(c) imaging the target using an imaging modality;

and wherein said locating is effected by steps including comparing at least one datum related to an image acquired using said combined tracking and imaging modality with said image of said region that is estimated to include the target.

62. The method of claim 61, wherein said imaging modality is a combined imaging and tracking modality that is used to effect said tracking.

63. The method of claim 44, further comprising the step of:

(c) during said tracking, acquiring an image of the target;

and wherein said locating is effected by steps including cross-correlating at least a portion of said image of the target with said at least portion of said image of said region.

64. The method of claim 44, further comprising the steps of:

(c) during said tracking, acquiring a plurality of images of the target; and

(d) combining said plurality of images of the target {e.g. by averaging} to provide a combined image of the target;

and wherein said locating is effected by steps including cross-correlating at least a portion of said combined image of the target with said at least portion of said image of said region.

65. A method of monitoring at least one target, comprising the steps of:

(a) tracking a first target, thereby obtaining an estimated track of said first target;

(b) acquiring an image of a first region that includes said first target;

(c) associating said estimated track with said image of said first target;

(d) acquiring an image of a second region; and

(e) comparing at least one datum related to said image of said first region to at least one datum related to said image of said second region to determine whether said first target is depicted in said image of said second region.

66. The method of claim 65, wherein said tracking of said first target, said acquiring of said image of said first region and said associating of said image of said first region with said estimated track are effected using a combined imaging and tracking modality.

67. The method of claim 66, wherein said combined imaging and tracking modality includes video motion detection.

68. The method of claim 65, wherein said image of said second region is acquired prior to said tracking of said first target.

69. The method of claim 65, wherein said image of said second region is acquired while tracking a second target using a combined imaging and tracking modality, and wherein said comparing is to determine whether said first and second targets are identical.

70. The method of claim 69, wherein said combined imaging and tracking modality includes video motion detection.

71. The method of claim 65, wherein said at least one datum related to said image of said first region is at least a portion of said image of said first region that depicts said first target, and wherein said at least one datum related to said image of said second region is at least a portion of said image of said second region that depicts a second target.

72. The method of claim 71, wherein said comparing is effected by steps including cross-correlating said at least portion of said image of said first region with said at least portion of said image of said second region.

73. A method of imaging a target, comprising the steps of:

(a) obtaining an image of a region that includes the target;

(b) determining coordinates of the target;

(c) based at least in part on said coordinates, aiming a combined imaging and tracking modality at the target;

(d) tracking and imaging the target, using said combined imaging and tracking modality; and

(e) based at least in part on said tracking by said combined imaging and tracking modality, extracting, from said image, a portion of said image that depicts the target.

74. The method of claim 73, wherein said obtaining is effected by said combined imaging and tracking modality during said tracking and imaging of the target.

75. The method of claim 73, wherein said determining of said coordinates is effected using a tracking modality.

76. The method of claim 75, wherein said tracking modality and said combined imaging and tracking modality produce respective estimated tracks of the target, the method further comprising the step of:

(f) associating said estimated tracks.

77. The method of claim 76, wherein said tracking modality and said combined imaging and tracking modality use different respective coordinate systems, and wherein said associating includes transforming said coordinates from said coordinate system of said tracking modality to said coordinate system of said combined imaging and tracking modality.

78. The method of claim 76, wherein said tracking modality and said combined imaging and tracking modality use different respective coordinate systems, and wherein said associating includes transforming said coordinates from said coordinate system of said combined imaging and tracking modality to said coordinate system of said tracking modality.

79. The method of claim 73, further comprising the step of:

(f) determining whether the target is moving, said aiming, tracking, imaging and extracting being effected only if the target is moving.

80. The method of claim 73, wherein said combined imaging and tracking modality includes video motion detection.

81. The method of claim 73, further comprising the step of:

(f) moving said combined imaging and tracking modality, in accordance with said coordinates, to a vantage point that facilitates said aiming.

82. The method of claim 81, wherein said moving is effected by steps including moving a platform whereon said imaging modality is mounted.

83. The method of claim 81, wherein said tracking of the target estimates an estimated track of the target, and wherein said vantage point is selected at least in part in accordance with said estimated track.

84. The method of claim 81, wherein said vantage point is selected at least in part in accordance with a location of at least one object that partly hides the target.

85. A system for imaging a target, comprising:

(a) a tracking subsystem for tracking the target, thereby obtaining a first estimated track of the target, said first estimated track including coordinates of the target;

(b) a combined imaging and tracking subsystem for imaging, according to said coordinates, a region that includes the target, and for then tracking the target, thereby providing a second estimated track of the target; and

(c) an extraction mechanism for extracting, from said image, a portion of said image that depicts the target, said extracting being based at least in part on said second estimated track.

86. The system of claim 85, further comprising:

(d) an association mechanism for associating said first and second estimated tracks.

87. The system of claim 85, further comprising:

(d) at least one vehicle on which said tracking subsystem and said combined imaging and tracking subsystem are mounted.

88. The system of claim 87, comprising a plurality of said vehicles, with said tracking subsystem and said combined imaging and tracking subsystem being mounted on respective said vehicles.

89. The system of claim 88, further comprising:

(e) a wireless mechanism for sending said first track from said tracking subsystem to said combined imaging and tracking subsystem.

90. The system of claim 87, wherein said at least one vehicle is an aircraft.

91. The system of claim 85, wherein said combined imaging and tracking subsystem uses video motion detection to track and image the target.

92. A method of selectively monitoring a plurality of targets, comprising the steps of:

(a) imaging the targets, substantially simultaneously, thereby providing a respective image of each target;

(b) displaying said images collectively;

(c) based at least in part on visual inspection of said images, selecting one of the targets; and

(d) devoting a resource to said one target.

93. The method of claim 92, further comprising the step of:

(e) ranking the targets;

said displaying then being effected in accordance with said ranking.

94. The method of claim 93, wherein said ranking is effected at least in part using automatic target recognition.

95. The method of claim 92, wherein said imaging is effected using at least one combined imaging and tracking modality during tracking of said targets.

96. The method of claim 92, wherein said resource includes a tracking modality for tracking said one target.

97. The method of claim 92, wherein said resource includes an imaging modality for further imaging of said one target.

98. The method of claim 92, wherein said resource includes a weapon for attacking said one target.

99. The method of claim 92, wherein said resource includes a display device for dedicated display of a location of said one target.

100. The method of claim 92, wherein said resource includes a mechanism for warning of a presence of said one target.

101. The method of claim 92, wherein said imaging is effected by steps including:

(i) acquiring at least one image of at least a portion of an arena that includes the targets; and

(ii) for each target, extracting, from said at least one image of said at least portion of said arena, a respective subportion, of said at least one image of said at least portion of said arena, that depicts said each target, said respective extracted subportion then being said respective image of said each target that is displayed.

102. A system for selectively monitoring a plurality of targets, comprising:

(a) at least one imaging modality for substantially simultaneously imaging the targets to provide a respective image of each target;

(b) a display mechanism for displaying said images collectively;

(c) a selection mechanism for selecting one of said images on said display mechanism; and

(d) a resource for devoting to said respective target of said selected image.