Inspection system and method for bond detection and validation of surface mount devices using sensor fusion and active perception

• US 6,269,179 B1
• Filed: 06/09/1998
• Issued: 07/31/2001
• Est. Priority Date: 05/31/1996
• Status: Expired due to Fees

• Current Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Original Assignee: Georgia Tech Research Corporation (University System of Georgia)
• Inventors: Vachtsevanos, George J., Dar, Iqbal M.
• Primary Examiner(s): Mancuso, Joseph
• Assistant Examiner(s): Bali, Vikkram
• Application Number:
  US09/093,685
• Time in Patent Office: 1,148 Days
• Field of Search: 382/141, 382/142, 382/143, 382/144, 382/145, 382/146, 382/147, 382/148, 382/149, 382/150, 382/224, 382/225, 382/226, 382/227, 382/228, 382/155, 382/156, 382/157 , 382/159, 382/160, 348/164, 348/87, 348/126, 348/131, 348/166, 348/165, 348/88, 356/51, 356/372, 356/237, 364/468.17, 364/557
• US Class Current: 382/149
• CPC Class Codes: G01N 2021/8867 using sequentially two or more inspection runs, e.g. coarse and fine, or detecting then analysing
• CPC Class Codes: G01N 21/88 Investigating the presence of flaws or contamination
• CPC Class Codes: G01N 21/95684 Patterns showing highly reflecting parts, e.g. metallic elements
• CPC Class Codes: G01N 25/72 Investigating presence of flaws
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/30152 Solder
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/0006 using a design-rule based approach

Abstract

A hybrid surface mount component inspection system which includes both vision and infrared inspection techniques to determine the presence of surface mount components on a printed wiring board, and the quality of solder joints of surface mount components on printed wiring boards by using data level sensor fusion to combine data from two infrared sensors to obtain emissivity independent thermal signatures of solder joints, and using feature level sensor fusion with active perception to assemble and process inspection information from any number of sensors to determine characteristic feature sets of different defect classes to classify solder defects.

Claims

1. A method for inspecting circuits by performing feature level sensor fusion using an active perception process, comprising the steps of:

using a gross inspection station to determine global circuit features and rapidly detect defects in the global circuit features, the gross inspection station configured to receive data from at least two infrared (IR) sensors and a vision sensor,

using a fine inspection station to generate local circuit features,

fusing the local circuit features using active perception;

performing data level sensor fusion at the gross inspection station by determining a ratio of the outputs of the two IR sensors thus canceling effectively the effect of sensor emissivity and improving data reliability;

using an off-line learning process at the gross inspection station, the off-line learning process configured to perform feature ordering for active perception, the feature ordering performed using a distinguishability measure and the processing time required to extract each feature;

evaluating a cost function for each feature;

using a perceptron classifier at the gross inspection station to classify defects;

computing a degree of certainty index at the gross inspection station the degree of certainty index representing a confidence level associated with a classification result;

performing feature level sensor fusion at the fine inspection station using an active perception process, the active perception process configured to fuse information from the IR and vision sensors and also configured to minimize the processing time by controlling the information gathering process by aligning features optimally so that features requiring less processing time and providing greater distinguishability between failure classes are listed first in the ordering scheme;

using an on-line learning process at the fine inspection station, the on-line learning process configured to adapt the active perception process to changes and disturbances by fine tuning the feature ordering; and

monitoring the system changes and disturbances using a measure called feature effectiveness that optimizes on-line the feature utilization in the classification task.

2. The method as defined in claim 1, further comprising the step of teaching the defects to the active perception process using an off-line learning process using a training data set.

3. The method as defined in claim 1, further comprising the step of teaching feature defects to the active perception process using an on-line learning process where, by monitoring the current state of system operation, the feature defects are rearranged such that the active perception process is adapted to the system'"'"'s current operational state.

4. The method as defined in claim 1, further comprising the step of ordering each of the features based upon their ability to distinguish between different defect classes.

5. The method as defined in claim 4, further comprising the step of quantifying the ability of each of the features to distinguish between different defect classes by determining an area of a defect region in which different defect classes overlap.

6. A system for inspecting circuits by performing feature level sensor fusion using an active perception process, comprising:

means for determining global circuit features and rapidly detecting defects in the global circuit features, the determining means configured to receive data from at least two infrared (IR) sensors and a vision sensor;

means for generating local circuit features;

means for fusing the local circuit features using active perception;

means for performing data level sensor fusion at the gross inspection station by determining a ratio of the outputs of the two IR sensors thus canceling effectively the effect of sensor emissivity and improving data reliability;

means for performing an off-line learning process at the gross inspection station, the off-line learning process configured to perform feature ordering for active perception the feature ordering performed using a distinguishability measure and the processing time required to extract each feature;

means for evaluating a cost function for each feature;

means for classifying defects using a perceptron classifier at the gross inspection station;

means for computing a degree of certainty index at the gross inspection station the degree of certainty index representing a confidence level associated with a classification result;

means for performing feature level sensor fusion at the fine inspection station using an active perception process, the active perception process configured to fuse information from the IR and vision sensors and also configured to minimize the processing time by controlling the information gathering process by aligning features optimally so that features requiring less processing time and providing greater distinguishability between failure classes are listed first in the ordering scheme, means for using an on-line learning process at the fine inspection station the on-line learning process configured to adapt the active perception process to changes and disturbances by fine tuning the feature ordering; and

means for monitoring the system changes and disturbances using a measure called feature effectiveness that optimizes on-line the feature utilization in the classification task.

7. The system as defined in claim 6, further comprising means for teaching the defects to the active perception process using an off-line learning process using a training data set.

8. The system as defined in claim 6, further comprising means for teaching the defects to the active perception process using an on-line learning process where, by monitoring the current state of system operation, the feature defects are rearranged such that the active perception process is adapted to the system'"'"'s current operational state.

9. The system as defined in claim 6, further comprising means for ordering each of the features based upon their ability to distinguish between different defect classes.

10. The system as defined in claim 9, further comprising means for quantifying the ability of each of the features to distinguish between the different defect classes by determining an area of a defect region in which the different defect classes overlap.

11. A computer readable medium having a program for inspecting circuits by performing feature level sensor fusion using an active perception process the medium comprising logic for performing the steps of:

using a gross inspection station to determine global circuit features and rapidly detect defects in the global circuit features the gross inspection station configured to receive data from at least two infrared (R) sensors and a vision sensor;

using a fine inspection station to generate local circuit features;

fusing the local circuit features using active perception;

performing data level sensor fusion at the gross inspection station by determining a ratio of the outputs of the two IR sensors, thus canceling effectively the effect of sensor emissivity and improving data reliability;

using an off-line learning process at the gross inspection station the off-line learning process configured to perform feature ordering for active perception, the feature ordering performed using a distinguishability measure and the processing time required to extract each feature;

evaluating a cost function for each feature;

using a perceptron classifier at the gross inspection station to classify defects;

computing a degree of certainty index at the gross inspection station, the degree of certainty index representing a confidence level associated with a classification result;

performing feature level sensor fusion at the fine inspection station using an active perception process, the active perception process configured to fuse information from the IR and vision sensors and also configured to minimize the processing time by controlling the information gathering process by aligning features optimally so that features requiring less processing time and providing greater distinguishability between failure classes are listed first in the ordering scheme;

using an on-line learning process at the fine inspection station, the on-line learning process configured to adapt the active perception process to changes and disturbances by fine tuning the feature ordering; and

monitoring the system changes and disturbances using a measure called feature effectiveness that optimizes on-line the feature utilization in the classification task.

12. The program as defined in claim 11, further comprising logic for performing the step of teaching the defect classes to the active perception process using an off-line learning process using a training data set.

13. The program as defined in claim 11, further comprising logic for performing the step of teaching feature defects to the active perception process using an on-line learning process where, by monitoring the current state of system operation, the feature defects are rearranged such that the active perception process is adapted to the system'"'"'s current operational state.

14. The program as defined in claim 11, further comprising logic for performing the step of ordering each of the features based upon their ability to distinguish between different defect classes.

15. The program as defined in claim 14, further comprising logic for performing the step of quantifying the ability of each of the features to distinguish between different defect classes by determining an area of a defect region in which different defect classes overlap.