Method and apparatus for inspecting a solder joint using a correlation neural network

• US 6,023,663 A
• Filed: 04/07/1997
• Issued: 02/08/2000
• Est. Priority Date: 04/08/1996
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Jong-Hyeong
• Primary Examiner(s): WACHSMAN, HAL D
• Application Number:
  US08/833,451
• Time in Patent Office: 1,037 Days
• Field of Search: 702/81, 702/34, 702/35, 702/40, 702/124, 702/126, 702/172, 702/183, 702/184, 702/189, 702/198, 250/559.34, 250/559.42, 250/559.43, 250/559.45, 324/717, 324/718, 228/179.1, 228/180.1, 228/180.21, 228/180.22, 364/468.17, 382/147, 382/149, 382/150, 382/156-160, 382/164, 382/165, 348/125, 348/126, 348/131, 348/133, 348/266, 348/269, 348/277, 356/237.1, 356/394, 356/384, 356/376, 706/20, 706/25, 706/41, 706/908, 706/916, 706/921, 706/40
• US Class Current: 702/81
• CPC Class Codes: G01N 21/95684 Patterns showing highly reflecting parts, e.g. metallic elements
• CPC Class Codes: G01R 31/309 of printed or hybrid circuits or circuit substrates
• CPC Class Codes: G01R 31/71 Testing of solder joints
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30152 Solder
• CPC Class Codes: G06T 7/0004 Industrial image inspection

Abstract

A solder joint inspection process and system includes using a correlation values to classify the shape of the solder joint. The solder joint is illuminated by a multiple colors, and the image of the solder joint is captured. This multiple color image is then converted into a plurality of one-dimensional vectors. Each one-dimensional vectors sequence is for one color of the multi-color illumination and for a one-dimensional spatial extent. Correlation values among all combinations of the one-dimensional vector sequences are computed and used for training an automatic solder joint shape classifier.

Claims

1. An apparatus for inspecting a solder joint, said apparatus comprising:

a solder joint image extracting means for extracting three-dimensional image information of a solder joint subjected to multi-color illumination; and

a solder joint shape classifying means for learning a classification hierarchy of solder joint images by using a training set of said solder joint images classified into said classification hierarchy by a human solder joint inspector, and for applying said learned classification hierarchy to said extracted image information, said solder joint shape classifying means including a preprocessor which converts said three-dimensional image information into a plurality of one-dimensional vector sequences, each one-dimensional vector sequence being for one color of the multi-color illumination and for a one-dimensional spatial extent, calculates correlation values among all combinations of said one-dimensional vector sequences, and supplies said correlation values as said training set.

2. An apparatus for inspecting a solder joint as claimed in claim 1, wherein said solder joint image extracting means comprises:

a multi-color illuminator for emitting at least three colors of light, each color at a different incident angled onto a solder joint inspection area;

a color camera for capturing color images reflected from said solder joint inspection area with a digital image pickup device; and

a color image grabber for grabbing a plurality of color frames, one frame for each of said colors, from an output of said color camera.

3. An apparatus for inspecting a solder joint as claimed in claim 2, wherein said solder joint shape classifying means comprises:

a correlation neural network for converting said plurality of color frames into said plurality of one-dimensional vector sequences, for calculating said correlation values among said vector sequences, for learning synaptic weights to apply to said correlation values to produce said classification hierarchy determined by said human solder joint inspector, and for applying previously learned said synaptic weights to said correlation values to produce an automatic classification within said classification hierarchy.

4. An apparatus for inspecting a solder joint as claimed in claim 3, wherein said correlation neural network comprises:

a trainable module for learning synaptic weights to apply to said correlation values to produce said classification hierarchy determined by said human solder joint inspector, and for applying previously learned said synaptic weights to said correlation values to produce an automatic classification within said classification hierarchy.

5. An apparatus for inspecting a solder joint as claimed in claim 2, wherein said multi-color illuminator comprises:

three circular, high-frequency neon lamps respectively emitting green light, red light, and blue light, which are coaxially tiered in sequence of decreasing diameter and in sequence of increasing distance from a plane containing said solder joint inspection area.

6. An apparatus for inspecting a solder joint as claimed in claim 5, wherein said green, red, and blue circular, high frequency neon lamps have diameters of about 130 mm, about 70 mm, and about 40 mm, respectively.

7. An apparatus for inspecting a solder joint as claimed in claim 5, wherein said green, red and blue circular high frequency neon lamps are positioned to have incident angles at said solder joint inspection area of about 70°

, about 40° and

about 20°

from an imaging axis of said color camera, respectively.

8. An apparatus for inspecting a solder joint, said apparatus comprising:

a color camera for capturing, with a digital image pickup device, color images reflected from said solder joint located in a solder joint inspection area;

three circular, high-frequency neon lamps, one emitting green light, the next emitting red light, and the last emitting blue light;

a color image grabber for grabbing a plurality of color frames, one frame for each of said plurality of colors, from an output of said color camera; and

,

a correlation neural network comprising a preprocessing module for calculating correlation values among a plurality of one-dimensional vector sequences derived from said plurality of color frames within one image, each one-dimensional vector sequence being for one color of the plurality of color frames and for a one-dimensional spatial extent, and a trainable module for learning synaptic weights to apply to said correlation values to produce a classification hierarchy determined by a solder joint inspector, and for applying previously learned said synaptic weights to said correlation values to produce an automatic classification within said classification hierarchy.

9. A method for inspecting a solder joint, comprising:

illuminating a solder joint with multiple-color light emitters distributed in three-dimensions;

extracting a multi-colored image of said solder joint;

determining a classification hierarchy of said solder joint by a human solder joint inspector;

classifying solder joint images by training a solder joint shape classifier to associate said extracted images of said solder joint with said classification hierarchy, and by applying said trained solder joint classifier to said extracted images of said solder joint, said classifying including converting said multi-colored image into a plurality of one-dimensional vector sequences, each one-dimensional vector sequence being for one color of the multi-color illumination and for a one-dimensional spatial extent, calculating correlation values among all combinations of said one-dimensional vector sequences, and using said correlation values for said training.

10. A method for inspecting a solder joint as claimed in claim 9, wherein said illuminating of a solder joint with multiple-color light in three dimensions comprises placing different color illuminators at different angles of incidence to a solder joint inspection area.

11. A method for inspecting a solder joint as claimed in claim 10, wherein said placement of different color illuminators at different angles of incidence to a solder joint inspection area comprises placing three circular light emitters of green, red, and blue light, coaxially in order of decreasing diameter and in sequence of increasing distance from said solder joint inspection area.

12. A method for inspecting a solder joint as claimed in claim 9, wherein said extracting of a multi-colored image of said solder joint comprises capturing a digital image of light reflected from a solder joint inspection area, and grabbing a plurality of color frames from said digital image.

13. A method for inspecting a solder joint as claimed in claim 9, wherein said classifying of solder joint images comprises operating a correlation neural network to learn to associate said extracted images of said solder joint with said classification hierarchy, and to act on said extracted images of said solder joint to produce an automatic classification within said classification hierarchy.

14. A method for inspecting a solder joint as claimed in claim 13, wherein operating of the correlation neural network comprises:

training synaptic weights of a trainable module to convert said correlation values into said classification hierarchy by comparing the network output to classifications made by said human solder joint inspector; and

applying the learned synaptic weights from said training to classify said multi-colored image of said solder joint within said classification hierarchy.

15. A method for inspecting a solder joint as claimed in claim 14, wherein said classifying further comprises:

(a) abridging said multi-colored image into an abridged image which has a size smaller than said multi-colored image, said abridged image including a plurality of columns of pixels;

(b) expressing each column of pixels of said abridged image as a set of one-dimensional vector sequences, the set including a one-dimensional vector sequence for each color of said multi-colored image, all of said sets of one-dimensional vector sequences comprising said plurality of one-dimensional vector sequences; and

(c) forming said correlation values by calculating cross-correlation and auto-correlation values among said one-dimensional vectors sequences.

16. A method for inspecting a solder joint as claimed in claim 15, wherein said training comprises:

(d) calculating a sample pattern error E_p and an average system error E by comparing network output values with target classification values defined by said classification hierarchy; and

(e) training said synaptic weights of said trainable module by using said sample pattern error and said average system error.

17. A method for inspecting a solder joint as claimed in claim 16, wherein in said calculating comprises,

activating the output O_j of a node j, where j indicates a layer of the correlation neural network in which the node is located, using sigmoid function as described by ##EQU16## where net_j is the net input to the node in layer j, the parameter σ

_j serves as a threshold or bias, and σ

_o is a shape parameter;

said sample pattern error E_p for each sample pattern is taken as ##EQU17## where t_p denotes said target classification values, and o_f denotes said network output values; and

said average system error E is defined as ##EQU18## where P is a total number of said multi-colored images in a training set, and subscript _p denotes each said multi-colored images in said training set.

18. A method for inspecting a solder joint as claimed in claim 16,

wherein said training comprises achieving convergence for said synaptic weights using a generalized-delta rule according to the following equation;

##EQU19## where η

is the learning rate, alpha is a momentum term, and W_kj is said synaptic weight for the output from network node k, where k indicates a layer of the correlation neural network in which the node is located, when used as input to network node j, where j indicates a layer of the correlation neural network in which the node is located.

19. A method for inspecting a solder joint as claimed in claim 15,

wherein said abridging includes setting one pixel of said abridged image to be equivalent to a window of a raw image, where said window is L pixels wide by K pixels high (L×

K), and said raw image is one color of said multi-colored image of said solder joint, and then said one pixel of said abridged image is described by ##EQU20## where, (u₀, v₀) is a position of an initial pixel of said raw image within said one pixel of said abridged image at position (i, j) of said abridged image, l is an integer indicating a column number in said raw image, k is an integer indicating a row number in said raw image, i is an integer indicating a column number in said abridged image, j is an integer indicating a row number in said abridged image, and τ

is a threshold value which is determined by a network designer;

wherein said expressing includes rewriting green, red and blue abridged images as said one-dimensional vector sequences which are denoted by

space="preserve" listing-type="equation">{G(i)|i=1, . . . , m}

space="preserve" listing-type="equation">{R(i)|i=1, . . . , m}

space="preserve" listing-type="equation">{B(i)|i=1, . . . , m}

wherein said forming includes calculating spatial correlation values of certain combinations of color vector sequences by using a correlation node, the output of said correlation node being described by ##EQU21## where {C₁ (i)|i=1, . . . , m} and {C₂ (i)|i=1, . . . , m} are the two vector sequences to be calculated, d is a spatial correlation distance between the two vector sequences, and m is the maximum number of elements in each of said vector sequences.