Method of and apparatus for optical imaging inspection of multi-material objects and the like

• US 6,603,877 B1
• Filed: 06/01/1999
• Issued: 08/05/2003
• Est. Priority Date: 06/01/1999
• Status: Expired due to Fees

• Current Assignee: B.E.L-Tronics, Inc. (Cedar Electronics Holdings Corp.)
• Original Assignee: B.E.L-Tronics, Inc. (Cedar Electronics Holdings Corp.)
• Inventors: Bishop, Robert
• Primary Examiner(s): Au, Amelia M.
• Assistant Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US09/323,441
• Time in Patent Office: 1,526 Days
• Field of Search: 382/165, 382/205, 382/224, 382/108, 382/228, 382/100, 382/141, 382/143, 382/145, 356/243--24, 356/239.3, 348/128, 348/131, 348/132
• US Class Current: 382/165
• CPC Class Codes: G01N 21/95607 using a comparative method
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30141 Printed circuit board [PCB]
• CPC Class Codes: G06T 2207/30148 Semiconductor; IC; Wafer
• CPC Class Codes: G06T 7/001 using an image reference approach

Abstract

A novel method all of and apparatus for categorizing different material regions of an object-to-be inspected, by optical scanning of the object to produce a pixel image thereof, discriminating different pixel regions of the image corresponding to possibly different materials on the basis of color, and texture brightness measurements of the pixel regions and assigning preliminary likelihoods to such discrimination through with ambiguities; and comparing the measurements of the pixel regions with their local neighborhood surroundings in the image to assist in resolving said ambiguities and determining the material categorizations of the pixel regions with a high likelihood.

Claims

1. A method of categorizing different material regions of an object-to-be-inspected, that comprises, optically scanning the object to produce a pixel image thereof;

discriminating different pixel regions of the image corresponding to possibly different materials on the basis of color and brightness measurements of the pixel regions and assigning preliminary likelihoods to such discrimination though with ambiguities; and

comparing the measurements of the pixel regions with their local neighborhood surroundings in the image to assist in resolving said ambiguities and to determine the materials categorization of the pixel regions with a higher likelihood.

2. The method of claim 1 wherein said pixel regions are also compared with a reference image further to aid in the resolution of said ambiguities.

3. The method of claim 1 wherein the pixel regions are also discriminated on the basis of texture of image regions.

4. The method of claim 3 wherein, in a highly textured area, discrimination is effected on the basis of average brightness measurement.

5. A method of categorizing different material regions of an object-to-be-inspected, that comprises, optically scanning the object to produce a pixel image thereof;

discriminating different pixel regions of the image corresponding to possibly different materials on the basis of color and brightness measurements of the pixel regions and assigning preliminary likelihoods to such discrimination though with ambiguities; and

comparing the measurements of the pixel regions with their local neighborhood surroundings in the image to assist in resolving said ambiguities and to determine the materials categorization of the pixel regions with a higher likelihood, wherein values of ambiguity are established to distinguish the different material regions from one another, the values being computed as the ratio of the second most likely assigned material likelihood to the most likely assigned material likelihood at that region.

6. The method of claim 5 wherein certainty values for region material likelihoods are assigned representing the highest likelihood for that region.

7. Apparatus for categorizing different material regions of an object-to-be-inspected having, in combination, an optical scanning camera system for inspecting and scanning successive regions of said object to produce pixel images thereof;

means for measuring the color and brightness of characteristics of the successive pixel image regions to discriminate possibly different materials thereof, assigning preliminary likelihoods of their being such different materials, with ambiguities; and

means for comparing the measurements of the pixel regions with their local neighborhood surroundings in the image and with a reference image, to assist in resolving said ambiguities and determining materials categorization of the pixel regions with a higher likelihood.

8. The apparatus claimed in claim 7 wherein color and brightness characteristics measuring means receives scanned image information from the pixel regions, and further measuring means is provided for processing information such as texture and average brightness characteristics from the surrounding local neighborhood of the regions and for comparing the same.

9. The apparatus claimed in claim 8 wherein said likelihood assigning is effected either before or after comparing the regions to their surrounding local neighborhood.

10. The apparatus claimed claim 9 wherein said comparing is effected by median filter means which, for each pixel, passes the median value of that pixel and its neighborhood measurements.

11. The apparatus claimed in claim 8 wherein means is provided for teaching and entering into the apparatus images of expected feature and material categories for reference in comparing with the likelihoods derived from brightness, color and texture characteristic measurements.

12. The apparatus claimed in claim 11 wherein means is provided for segmenting the image into material categories using the learned reference brightness, color and texture characteristics to determine the most likely material at each point of the image, classifying the material on the basis of the greatest likelihood that does not exceed ambiguity limits.

13. The apparatus claimed in claim 12 wherein the image segmenting means provides an image point intensely at each spatial location that is proportional to its measured likelihood, with the brighter the point, the higher the likelihood of the given material.

14. The apparatus claimed in claim 13 wherein means is provided for improving that accuracy of the segmentation process by providing measurements of certainty equal to the likelihood of the selected material, and ambiguity equal to the ratio of the second most likely material divided by the likelihood of the most probable material.

15. The apparatus claimed in claim 14 wherein means is provided for creating images of the certainty and ambiguity measurements for the segmented image, to aid and detect, such as an incorrect material that is greater than a predetermined minimum size at a given location.

16. The apparatus claimed in claim 15 wherein the object is a multilevel, highly textured metal mask having material categories of top and bottom metal surfaces, side wall transitions there-between, and holes through the material, the material likelihoods of each of which are obtained with certainty and ambiguity values.

17. The apparatus claimed in claim 7 wherein said pixel images are fed from the camera system to a material analysis card for processing to determine the measurements of said characteristics and to compare the same with a reference look-up table that determines and assigns said preliminary likelihoods.

18. The apparatus claimed in claim 7 wherein said scanning camera system comprises two cameras adapted to collect images of the object either from two different color filters or from two different focal planes.

19. The apparatus claimed claim 18 wherein a combine card is provided having a plurality of reference look-up tables with means for comparing said characteristics of the pixel images from both cameras and assigning material likelihoods those materials requiring information from both cameras, such as color and focal plane.

20. The apparatus of claim 19 wherein said combine card is provided with a further reference look-up table for enabling correlation of the likelihoods from each camera and the likelihoods determined jointly from both cameras and assigning preliminary likelihoods for each material based on both camera images together.

21. The apparatus of claim 20 wherein a material classification card is further provided for comparing likelihoods and ambiguities of the regions and their respective local neighborhood and for combining such information with predetermined information of said reference image, thereby to determine the material categorization of the region with higher likelihood.

22. The apparatus of claim 21 wherein means is provided for passing the material categorization information to defect monitoring circuitry.

23. The apparatus of claim 21 wherein the material analysis card comprises median filters for measuring image values over regions, and lookup tables used to combine measured values and composite likelihood values.

24. The apparatus of claim 23 wherein the filters are cascadable spaced filters adapted to process in arbitrarily large area, with delay lines associated with the look-up tables for enabling resynchronization.

25. The apparatus of claim 24 wherein image region texture is determined by means for calculating the absolute value of the difference between the brightness of a pixel in the image and the filtered average brightness around it.

26. The apparatus of claim 25 wherein said average brightness is obtained by means for locating areas of high texture and applying the same to a maximum filter and to a smoothing median filter.

27. The apparatus of claim 25 wherein said delay lines are adjustable and insure that all processed data inputted to look-up tables is spatially and temporarily aligned, with the original image pixel and the processed filtered output first being assigned preliminary likelihoods with look-up tables, and then assigning one value to each possible material at each pixel location.

28. The apparatus is claimed in claim 23 wherein the material classification card is provided with means for calculating the certainty and ambiguity of each image pixel and using such to determine the material for that pixel.

29. The apparatus as claimed in claim 28 wherein preliminary material likelihoods for each pixel are passed through a minimum filter to smooth away any single isolated pixel with a higher likelihood than that of its immediate neighbors.

30. The apparatus as claimed in claim 29 wherein the minimum filter likelihood outputs for all materials are applied to a first look-up table for determining a factor K which is a function of the pixel certainty minus its ambiguity, wherein both the pixel certainty and ambiguity are functions of the pixel likelihood valued.

31. The apparatus as claimed in claim 30 wherein the minimum filter likelihood outputs are also fed through low-pass filters, one per material, to determine the material likelihoods of the local neighborhood of each pixel.

32. The apparatus as claimed in claim 31 wherein means is provided for applying the local neighborhood likelihoods to a further look-up table to produce factors K for each pixel neighborhood.

33. The apparatus as claimed in claim 32 wherein delay means are provided to achieve spatial and temporal alignment to enable the apparatus to resynchronize all of the pixel likelihoods, the pixel K'"'"'s, neighborhood likelihoods and the neighborhood K'"'"'s, and to apply them together to another look-up table for out-putting the final material likelihoods for each pixel.

34. The apparatus of claim 33 wherein means is provided for combining the last-named table outputs of final material likelihoods with a prior information from a known reference image, and, with a further look-up table, comparing each pixel'"'"'s final material likelihoods with material classification of a known correct pixel, enabling classification of each pixel of the object into given material categories.

35. The apparatus of claim 34 wherein means is provided for supplying the material category information to defect inspection circuits.

36. The apparatus of claim 18 wherein the cameras are differently focused upon different regions of the object, such as top and bottom surfaces, and means is provided for simultaneously computing the differential image texture received by the two cameras for the regions.

37. The apparatus claimed in claim 36 wherein the object is provided with one or more holes and bottom illumination therethrough supplementing top illumination for the two-camera imaging, and means is provided for differentiating glint images through imaging holes dark on the camera focused on the top surface and bright on the bottom surface.

38. The apparatus claimed in claim 36 wherein means is provided for detecting boundary transition regions such as side walls between the top and bottom surfaces of the object by measuring the brightness and texture of such transition regions to determine dark areas of low texture that characterizes the images of such regions.

39. Apparatus for categorizing different material regions of an object-to-be-inspected having, in combination, an optical scanning camera system for inspecting and scanning successive regions of said object to produce pixel images thereof;

means for measuring the color and brightness of characteristics of the successive pixel image regions to discriminate possibly different materials thereof, assigning preliminary likelihoods of their being such different materials, with ambiguities; and

means for comparing the measurements of the pixel regions with their local neighborhood surroundings in the image and with a reference image, to assist in resolving said ambiguities and determining materials categorization of the pixel regions with a higher likelihood, wherein said scanning camera system comprises two cameras adapted to collect images of the object either from two different color filters or from two different focal planes, and wherein a combine card is provided having a plurality of reference look-up tables with means for comparing said characteristics of the pixel images from both cameras and assigning material likelihoods to those materials requiring information from both cameras, such as color and focal plane, wherein said combine card is provided with a further reference look-up table for enabling correlation of the likelihoods from each camera and the likelihoods determined jointly from both cameras and assigning preliminary likelihoods for each material based on both camera images together; and

wherein a material classification card is further provided for comparing likelihoods and ambiguities of the regions and their respective local neighborhoods and for combining such information with predetermined information of said reference image, thereby to determine the material categorization of the region with higher likelihood, and wherein the material analysis card comprises median filters for measuring image values over regions, and lookup tables used to combine measured values and composite likelihood values, the filters being cascadable spaced filters adapted to process in arbitrarily large area, with delay lines associated with look-up tables for enabling resynchronization.