Method and apparatus for 3-dimensional vision and inspection of ball and like protrusions of electronic components

• US 8,885,040 B2
• Filed: 03/30/2007
• Issued: 11/11/2014
• Est. Priority Date: 06/21/2006
• Status: Active Grant

• Current Assignee: Mit Semiconductor Company Limited
• Original Assignee: Generic Power Pte., Ltd. (Manufacturing Integration Technology Ltd. /2 Subsidiaries/)
• Inventors: Puah, Yong Joo, Tang, Hak Wee, Teo, Soon Poo
• Primary Examiner(s): Murray, Daniel C
• Application Number:
  US12/305,835
• Publication Number:
  US 20100328435A1
• Time in Patent Office: 2,783 Days
• Field of Search: 348/47, 382/147, 382/150, 702/81
• US Class Current: 348/126
• CPC Class Codes: G01B 11/0608 Height gauges
• CPC Class Codes: G01N 21/8806 Specially adapted optical and illumination features
• CPC Class Codes: G01N 21/951 Balls
• CPC Class Codes: G01N 21/95684 Patterns showing highly reflecting parts, e.g. metallic elements
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30148 Semiconductor; IC; Wafer
• CPC Class Codes: G06T 2207/30152 Solder
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/0006 using a design-rule based approach
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H05K 13/0818 Setup of monitoring devices prior to starting mounting operations; Teaching of monitoring devices for specific products; Compensation of drifts during operation, e.g. due to temperature shifts

Abstract

A method for 3-dimensional vision inspection of objects, such as microelectronic components, which have protrusions as features to be inspected, such as input/output contact balls, is disclosed. The method comprises the steps of determining interior and exterior parameters of a stereo camera system, forming a rectified stereo camera system with a rectified camera coordinate system from the parameters of the stereo cameras determined above. A pair of images of the object having a plurality of co-planar protrusions on one surface is captured by the stereo cameras system and is transformed into the rectified coordinate system to form a pair of rectified images. Conjugate points of each protrusion are then determined in the rectified images for the measurement of its location, co-planarity and height. Various configurations of the apparatus for capturing the images and processing the image data are also disclosed.

Claims

1. A method for 3-dimensional vision inspection of objects, including microelectronic components, having protrusions as features to be inspected, wherein said protrusions having non-planar surfaces and/or surface curvatures, said method comprising the steps of:

(a) providing a stereo camera system comprising at least a pair of cameras arranged in a stereovision configuration, wherein the first camera captures a side view of said object, and the second camera captures an opposite side view of said object;

(b) calibrating said stereo camera system to determine the interior parameters of said respective cameras and exterior parameters, or relative pose including position and orientation, of the second camera with respect to the first camera, wherein the interior parameters include focal length, optics center, distortion coefficients, and sensor cell size and the exterior parameters or the relative pose includes roll angle, tilt angle, pan angle, and translation along X axis, Y axis, and Z axis respectively of one camera with respect to the other camera;

(c) forming a rectified stereo camera system with a rectified camera coordinate system from the parameters of said real stereo camera system determined above, and determining the interior parameters of said respective rectified cameras and exterior parameters, or relative pose including position and orientation, of the second rectified camera with respect to the first rectified camera, wherein, the optical centers of the rectified cameras are the same as those of said stereo cameras, but the rectified cameras are rotated such that both rectified cameras are in parallel and their X axes are collinear, and the orientation of the rectified cameras is defined by the cross product of line connecting the optical centers of said stereo cameras, and line of intersection of the image planes of said stereo cameras, and both rectified cameras have the same focal length;

(d) using said stereo camera system to capture a pair of images of said object comprising two side views of surface curvatures of each of the protrusions of said object;

(e) rectifying said pair of images to produce a pair of rectified images as captured by said rectified stereo camera system, wherein, the rectified images are on the image planes of the rectified cameras;

(f) determining a pair of conjugate points for the tip of each protrusion in the rectified images; and

(g) determining the location of each protrusion, which include x, y, and z coordinates, in the rectified camera coordinate system based on said pair of conjugate points and said interior and exterior parameters of said rectified stereo camera system for 3-dimensional vision inspection of said objects.

2. The method for 3-dimensional vision inspection according to claim 1 further including the step of:

(h) calculating the co-planarity of the protrusions based on the locations of the protrusions by forming a seating plane using a standard best-fit plane algorithm.

3. The method for 3-dimensional vision inspection according to claim 2 further including the step of:

(i) calculating the height of each protrusion wherein a substrate plane is determined and the perpendicular distance of protrusion to the substrate plane is taken as the height of the protrusion.

4. The method for 3-dimensional vision inspection according to claim 3 wherein the height of the protrusion is calculated as follows:

(i) approximating a plane, based on the locations of the protrusions having co-planarity values;

(ii) shifting said plane by the amount of nominal protrusion height to form the substrate plane; and

(iii) taking the height of the protrusion as the perpendicular distance of said protrusion to the substrate plane.

5. The method for 3-dimensional vision inspection according to claim 1 wherein the calibration step is-achieved with the use of a calibration grid.

6. The method for 3-dimensional vision inspection according to claim 1 wherein the conjugate points of each protrusion being determined in the step (f) includes

(i) locating in each rectified image, the edges of the object to draw up the boundaries of the object;

and

(ii) performing standard blob analysis to locate the gross position of each protrusion.

7. The method for 3-dimensional vision inspection according to claim 6 wherein step (f) further includes

(iii) projecting two edge-detecting windows onto each protrusion to locate its top and bottom edges up to sub-pixel accuracy;

(iv) projecting a third edge-detecting window from a central vertical position of the protrusion to detect the left edge of the protrusion in the rectified right image and the right edge of the protrusion in the rectified left image up to sub-pixel accuracy; and

(v) assigning conjugate points to be the right edge detected in the rectified left image and the left edge detected in the rectified right image of the protrusion.

8. The method for 3-dimensional vision inspection according to claim 1 wherein step (g) further includes

(i) determining the location of the protrusion in the rectified camera coordinate system utilizing the positions of the conjugate points and the interior parameters and exterior parameters, or relative pose of the rectified cameras computed during calibration;

or

(ii) calculating disparity of the protrusion to be the difference between the positions of the conjugate points; and

(iii) determining the location of the protrusion in the rectified camera coordinate system utilizing the position of one of the conjugate points, the disparity, and the interior parameters and exterior parameters, or relative pose of the rectified cameras computed during calibration.

9. An apparatus for 3-dimensional vision inspection of non-planar objects, including microelectronic components, wherein the method according to claim 1 is enabled for calibration and/or measurement purposes, said apparatus comprising:

(a) at least a pair of cameras arranged in a stereovision configuration, wherein the first camera captures a side view of said object, and the second camera captures an opposite side view of said object;

(b) a pair of reflective surfaces for reflecting said object into each of said cameras respectively; and

(c) means for mounting said object in the field of view of said cameras;

wherein said object has a plurality of co-planar protrusions on at least one surface or substrate.

10. The apparatus for 3-dimensional vision inspection according to claim 9 wherein the object to be inspected is placed within a ring of light source and is illuminated thereby.

11. The apparatus for 3-dimensional vision inspection according to claim 10 wherein the light source comprises a plurality of light-emitting diodes (LED) with their light beam positioned at an inclination of 25°

towards the object to be inspected.

12. The apparatus for 3-dimensional vision inspection according to claim 11 wherein a background light source is further provided.

13. The apparatus for 3-dimensional vision inspection according to claim 12 wherein the cameras are connected to a data processor via a frame grabber or a high-speed data transfer bus including an IEEE-1394 bus (also known as “

Firewire”

).

14. The apparatus for 3-dimensional vision inspection according to claim 13 wherein the data processor is a computer capable of running at least a program for calibrating, rectifying and processing captured images.

15. The apparatus for 3-dimensional vision inspection according to claim 9 further including a third camera.

16. The apparatus for 3-dimensional vision inspection according to claim 15 wherein the third camera is placed in between the pair of first two cameras and is configured for inspecting another type of feature of the object, or another type of object.

17. The apparatus for 3-dimensional vision inspection according to claim 9 wherein a calibration reticle for having its image captured for calibration process is provided removably mounted.

18. The apparatus for 3-dimensional vision inspection according to claim 9 wherein the reflective surfaces are positioned at an inclination of 22.5°

from parallel axis towards the object to be inspected so that said object is reflected into the cameras.