Hierarchical information fusion object recognition system and method

• US 5,963,653 A
• Filed: 06/19/1997
• Issued: 10/05/1999
• Est. Priority Date: 06/19/1997
• Status: Expired due to Term

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Chen, Yang, Doria, David M., Reiser, Kurt, McNary, Charles, Webster, David W.
• Primary Examiner(s): Chang, Jon
• Assistant Examiner(s): Dastouri, Mehrdad
• Application Number:
  US08/878,863
• Time in Patent Office: 838 Days
• Field of Search: 382/100, 382/103, 382/190, 382/228, 342/62, 342/64, 342/104, 342/109
• US Class Current: 382/103
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 7/417 involving the use of neural networks
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes

Abstract

A hierarchical object recognition method for aggregation, interpretation and classification of information from multiple sensor sources on the detection feature attribute level. The system extracts information derived from each sensor source to obtain detections and their feature attributes. At least two processing streams, one for each sensor source, are provided for converting the detections and their feature attributes into hypotheses on identity and class of detected objects. The detections are shared and combined between the two processing streams using hierarchical information fusion algorithms to determine which ones of the hypotheses on identity and class of detected objects have sufficient probabilities for classifying the information.

Claims

1. A hierarchical object recognition method for aggregation, interpretation and classification of information from multiple sensor sources on the detection feature attribute level, comprising the following steps:

extracting information derived from each said sensor source to obtain detections and their feature attributes;

providing at least two processing streams, one for each said sensor source, for converting said detections and their feature attributes into hypotheses on identity and class of detected objects;

fusing said hypotheses at different feature levels for each said processing streams via hierarchical information fusion algorithms; and

sharing and combining the detection feature attributes and hypotheses about the detections between the two processing streams using said hierarchical information fusion algorithms to determine which ones of said hypotheses on identity and class of detected objects have the highest probabilities, and providing said fused hypotheses to said processing streams for determining hypotheses based on said feature attributes.

2. The method as claimed in claim 1, wherein:

each said processing stream has a plurality of processing stages, and

said hierarchical information fusion algorithms are performed at each said processing stage with decision nets based on evidential reasoning paradigms, to restrict the hypotheses on identity and class of detected objects and reduce the probability of misclassification of detected objects.

3. The method as claimed in claim 2, wherein the decision nets are project-defined and knowledge-driven decision nets, selected from the group of Markov tree systems and neural networks.

4. The method as claimed in claim 3, further comprising the step of:

generating and updating the decision nets automatically from a problem description, and wherein

the decision nets are made in modular form to allow easy and rapid updates.

5. The method as claimed in claim 1, wherein the method is adapted for a hierarchical real-time SAR/FLIR air-to-ground targeting object recognition system for automated target classification on the detection feature attribute level,

each said processing stream has a plurality of processing stages,

the sensors are a synthetic aperture radar (SAR) and a forward-look infra-red (FLIR),

said processing stages are SAR processing stages and FLIR processing stages,

further comprising:

a spatial registration module for spatial registration of the detections from said multiple sensor sources to establish correspondences of common objects and to reject uncorrelated clutter detections, wherein the sensor sources include on-board and off-board sensor sources; and

an off-board cues module for accepting off-board cues including information about target type, target position, target orientation, target velocity, and communication/radiation signatures.

6. The method as claimed in claim 5, wherein:

said hierarchical information fusion algorithms are performed at each said processing stage with decision nets based on evidential reasoning paradigms, to restrict the hypotheses on identity and class of detected objects and reduce the probability of misclassification of detected objects, wherein at least one of said processing stages receives from said fusion algorithms the restricted hypotheses for feature processing, and

the decision nets are project-defined and knowledge-driven decision nets, selected from the group of Markov tree systems and neural networks.

7. The method as claimed in claim 6, further comprising the step of:

generating and updating the decision nets automatically from a problem description, and wherein

the decision nets are made in modular form to allow easy and rapid updates.

8. A hierarchical object recognition method for aggregation, interpretation and classification of information from multiple sensor sources on the detection feature attribute level, wherein the method is adapted for a hierarchical real-time SAR/FLIR air-to-ground targeting object recognition system for automated target classification on the detection feature attribute level, comprising the following steps:

extracting information derived from each said sensor source to obtain detections and their feature attributes;

providing at least two processing streams, one for each said sensor source, for converting said detections and their feature attributes into hypotheses on identity and class of detected objects;

providing a plurality of fusion engines having hierarchical information fusion algorithms for fusing said hypotheses at different feature levels for each said processing streams; and

sharing and combining the detection feature attributes and hypotheses about the detections between the two processing streams using hierarchical information fusion algorithms to determine which ones of said hypotheses on identity and class of detected objects have the highest probabilities, and at least one of said fusion engines providing said fused hypotheses of said one fusion engine to said processing streams for determining hypotheses based on said feature attributes.

9. The method as claimed in claim 8, wherein:

each said processing stream has a plurality of processing stages,

the sensors are a synthetic aperture radar (SAR) and a forward-look infra-red (FLIR),

said processing stages are SAR processing stages and FLIR processing stages,

further comprising;

a spatial registration module for spatial registration of the detections from said multiple sensor sources to establish correspondences of common objects and to reject uncorrelated clutter detections, wherein the sensor sources include on-board and off-board sensor sources; and

an off-board cues module for accepting off-board cues including information about target type, target position, target orientation, target velocity, and communication/radiation signatures.

10. The method as claimed in claim 9, wherein:

said hierarchical information fusion algorithms are performed at each said processing stage with decision nets based on evidential reasoning paradigms, to restrict the hypotheses on identity and class of detected objects and reduce the probability of misclassification of detected objects.

11. The method as claimed in claim 10, wherein the decision nets are project-defined and knowledge-driven decision nets, selected from the group of Markov tree systems and neural networks.

12. The method as claimed in claim 11, further comprising the step of:

generating and updating the decision nets automatically from a problem description, and wherein

the decision nets are made in modular form to allow easy and rapid updates.

13. A hierarchical object recognition system for aggregation, interpretation and classification of information from multiple sensor sources on the detection feature attribute level, said system comprising:

an extraction information engine derived from each said sensor source using hierarchical information extraction algorithms to obtain detections and their feature attributes;

at least two processing streams, one for each said sensor source, each said processing stream having a plurality of processing stages adapted for converting said detections and their feature attributes into hypotheses on identity and class of detected objects; and

a fusion engine having a fuse decision module for sharing and combining the detection feature attributes and hypotheses about the detections between the at least two processing streams using hierarchical information fusion algorithms to determine which ones of said hypotheses on identity and class of detected objects have the highest probabilities,

said fusion engine sharing and combining the hypotheses of at least one of said processing stages for each said sensor source in order to provide the combined hypotheses to another one of said processing stages for determining hypotheses based on said feature attributes.

14. The system as claimed in claim 13, wherein:

each said processing stream comprising;

an area of interest stage,

an indexer stage, wherein at least one of said indexer stages generates feature data based upon data provided by at least one of said fusion engines and by another indexer stage,

a spatial features stage, and

an internal features stage,

the extraction information engine has a fuse detections stage which confirms the objects detected by each said processing stream, the fuse decision module comprises at least one fuse hypotheses stage for fusing aggregated detections from each processing stream to restrict the set of hypotheses for the subsequent stages of processing, and

the area of interest stage includes;

a search/detect stage,

a geometric features extraction stage, and

a reflective features extraction stage.

15. The system as claimed in claim 13, wherein said hierarchical information fusion algorithms are performed at each said processing stage with decision nets based on evidential reasoning paradigms, to restrict the hypotheses on identity and class of detected objects and reduce the probability of misclassification of detected objects.

16. The system as claimed in claim 15, wherein the decision nets are project-defined and knowledge-driven decision nets, selected from the group of Markov tree systems and neural networks, generated and updated automatically from a problem description, and made in modular form to allow easy and rapid updates.

17. The system as claimed in claim 13, wherein:

said system is a hierarchical real-time SAR/FLIR air-to-ground targeting object recognition system for automated target classification on the detection feature attribute level,

the sensors are a synthetic aperture radar (SAR) and a forward-look infra-red (FLIR),

said processing stages are SAR processing stages and FLIR processing stages,

further comprising;

a spatial registration module for spatial registration of the detections from said multiple sensor sources to establish correspondences of common objects and to reject uncorrelated clutter detections, wherein the sensor sources include on-board and off-board sensor sources; and

off-board cues module for accepting off-board cues including information about target type, target position, target orientation, target velocity, and communication/radiation signatures.

18. The system as claimed in claim 17, wherein said hierarchical information fusion algorithms are performed at each said processing stage with decision nets based on evidential reasoning paradigms, to restrict the hypotheses on identity and class of detected objects and reduce the probability of misclassification of detected objects.

19. The system as claimed in claim 18, wherein the decision nets are project-defined and knowledge-driven decision nets, selected from the group of Markov tree systems and neural networks, generated and updated automatically from a problem description, and made in modular form to allow easy and rapid updates.

20. The system as claimed in claim 19, wherein:

the SAR subsystem comprising;

a radar area of interest stage,

a radar indexer stage,

a radar spatial features stage, and

a radar internal features stage,

the FLIR subsystem comprising;

a FLIR area of interest stage,

a FLIR indexer stage,

a FLIR spatial features stage, and

a FLIR internal features stage,

the extraction information engine has a fuse detections stage which confirms the objects detected by both the SAR and the FLIR subsystem,

the fuse decision module comprises at least one fuse hypotheses stage for fusing aggregated detections from the SAR and FLIR subsystem to restrict the set of hypotheses for the subsequent stages of processing,

the radar area of interest stage includes;

a radar search/detect stage,

a radar geometric features extraction stage, and

a radar reflective features extraction stage, and

the FLIR area of interest stage includes;

a FLIR search/detect stage,

a FLIR geometric features extraction stage, and

a FLIR emissive features stage.

21. A hierarchical information fusion target recognition system for classifying targets based upon target data from a plurality of sensor sources:

an information extraction engine for processing target data from each of said sensor sources at different feature levels in order to formulate target hypotheses regarding classification of said targets;

a plurality of fusion engines connected to said information extraction engine for fusing target hypotheses at each of said different feature levels from said sensor sources, and at least one of said fusion engines providing said fused target hypotheses of said one fusion engine to said information extraction engine for determining hypotheses on identity and class of detected objects; and

a target list generator connected to said fusion engines for generating a list of targets based upon said fused target hypotheses.

22. The hierarchical information fusion target recognition system of claim 21 wherein said different feature levels include spatial features and internal features related to said target.

23. The hierarchical information fusion target recognition system of claim 22 wherein said internal features relate to reflective and emissive features of said targets.

24. The hierarchical information fusion target recognition system of claim 21 further comprising:

a detections fusion engine for fusing detections of said sensor sources in order to detect false alarms in classifying targets, said detections fusion engine performing said detections based upon correlating area of interest data from said sensor sources, said detections fusion engine providing said fused detections to said information extraction engine for processing target data at a predetermined feature level.

25. The hierarchical information fusion target recognition system of claim 21 further comprising:

first and second indexer stages being respectively associated with a first and second sensor source for formulating first and second hypotheses regarding target classification based respectively upon said target data of said first and second sensor sources,

said first indexer stage generating first geometric feature data based upon target data from said first sensor,

said first indexer stage providing said first geometric data to said second indexer stage,

said second indexer stage generating second geometric feature data based upon target data from said second sensor and upon said provided first geometric data,

said second indexer stage formulating said second hypotheses regarding target classification of said target data from said second sensor source; and

a hypotheses fusion engine connected to said first and second indexer stages for fusing said first and second formulated hypotheses,

wherein said target list generator classifies targets based upon said fused first and second hypotheses.

26. The hierarchical information fusion target recognition system of claim 25 further comprising:

a detections fusion engine for fusing detections of said sensor sources in order to detect false alarms in classifying targets, said detections fusion engine performing said detections based upon correlating area of interest data from said sensor sources, said detections fusion engine providing said fused detections to said information extraction engine for processing target data at a predetermined feature level,

said first and second indexer stages utilizing said generated fused detections from said detections fusion engine in order to formulate said first and second hypotheses.

27. The hierarchical information fusion target recognition system of claim 25 further comprising:

a joint target model database for storing model geometric features of predetermined types of targets,

said first and second indexer stages comparing said determined first and second geometric feature data with said stored geometric model features in order to formulate first and second hypotheses.

28. The hierarchical information fusion target recognition system of claim 25 further comprising:

first and second spatial features stages being respectively associated with a first and second sensor source,

said first and second spatial features stages formulating first and second spatial hypotheses regarding target classification based respectively upon said target data of said first and second sensor sources and upon said fused first and second hypotheses from said first and second indexer stages; and

;

a spatial features hypotheses fusion engine connected to said first and second spatial features stages for fusing said first and second formulated spatial hypotheses,

wherein said target list generator classifies targets based upon said fused first and second spatial hypotheses.

29. The hierarchical information fusion target recognition system of claim 21 wherein said fusion engines include decision nets which utilize evidential reasoning paradigms in order to fuse hypotheses regarding said target data.