Machine vision system for position sensing

• US 4,891,767 A
• Filed: 06/02/1988
• Issued: 01/02/1990
• Est. Priority Date: 06/02/1988
• Status: Expired due to Fees

• Current Assignee: Combustion Engineering Incorporated (ABB Limited)
• Original Assignee: Combustion Engineering Incorporated (ABB Limited)
• Inventors: Allen, Matthew D., Neuschaefer, Carl H., Williams, Thomas D., Glazer, Frank, Kowles, Joseph E., Rzasa, Peter W.
• Primary Examiner(s): MACDONALD, ALLEN R
• Application Number:
  US07/201,608
• Time in Patent Office: 579 Days
• Field of Search: 364/513, 364/188, 364/189, 364/190, 318/571, 901/2, 901/16, 901/9, 901/30
• US Class Current: 700/259
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: B25J 9/1697 Vision controlled systems

Abstract

The video image of the end effector (26) and work surface (32) in the vicinity of the end effector is converted into a series of continually generated digitized images (132), which are further processed to detect and track the relative motion between the end effector and the work surface. After calibration (126), whereby an index of coordinates and arm orientation is established for a known positioning of the end effector relative to a known reference feature (28a) on the work surface, the tracking (168) of the motion of the end effector is utilized to continually update the index (104) to indicate the actual coordinate and orientation of the end effector relative to the work surface. The updated index is displayed on a live video monitor (44), and can be used as input to the feedback circuit (94) of the manipulator control system. In another embodiment of the invention, a high resolution monitor (52) displays an image of substantially the entire work surface, and particularly the features of interest, in the nature of a work surface map, and superimposes thereon a visual indication of the location, orientation and movement of the end effector, using the updated index information.

Claims

1. A system for sensing the position of an end effector on a remotely manipulated robot arm, comprising:

a manipulator arm having at least two degrees of freedom for displacing an end effector;

an arm control system responsive to input signals generated remotely from the arm, for positioning the arm relative to a stationary work surface;

means for illuminating the work surface in the vicinity of the end effector;

means for generating a video image of the end effector and the illuminated surface;

means for establishing a reference coordinate of the end effector for a known reference position of the arm relative to the work surface;

means for generating a digitized image of at least a portion of the generated video image when the video image includes the reference position;

means for calibrating the digitized image to the video image by associating the reference coordinate of the end effector in the video image with a reference coordinate of the end effector in the digitized image;

means for continually processing changes in the digitized images resulting from changes in the video images accompanying displacement of the arm from the reference position by the arm control system, to continually track the path of movement of the arm; and

means responsive to the means for processing changes, for continually generating an index indicative of the actual coordinate of the end effector relative to the work surface as the arm is displaced along the work surface.

2. The system of claim 1, further including means coupled to the means for processing, for storing a data file representing preselected geometric features of the work surface.

3. The system of claim 2, wherein the work surface is a steam generator tube sheet and the preselected features include the tubes in the tube sheet.

4. The system of claim 3, wherein the preselected features further include a reference feature which is discernible in said generated video image when the arm is in said reference position.

5. The system of claim 3, wherein,

the means for generating a video image is coupled to the means for storing a data file of the work surface and to the means for generating a digitized image,

the means for continually generating an index includes means for generating an index associated with each of a plurality of tubes that are visible in the video image, and

the system further includes means for displaying the respective indices of the images of each of said plurality of the tubes that are visible in the video image.

6. The system of claim 2, further including a high resolution monitor coupled to the means for storing the data file and the means for continually processing the changes in digitized images, for generating a video map of the entire work surface and indicating on the map the current position of the end effector relative to the work surface.

7. The system of claim 1, further including means coupled to the means for generating an index, for displaying the index in the video image.

8. The system of claim 1, wherein

the work surface has a plurality of significant discrete features, and

the means for continually processing includes means for converting the digitized image of each of at least two features into at least one salient attribute and means for comparing changes in the salient attributes resulting from changes in the video images accompanying displacement of the arm, to track the path of movement of the arm.

9. The system of claim 8, wherein,

the work surface is a steam generator tube sheet,

the significant discrete feature is an individual tube end, and

the salient attribute of the tube end includes the centroid of the tube end.

10. The system of claim 1, wherein the arm control system includes,

sensor means coupled to the means for continually generating an index, for generating a first signal commensurate with the actual displacement of the arm in each direction of freedom,

target means for generating a second signal, indicative of the desired displacement of the arm in each direction of freedom, and

feedback means responsive to the first and second signals, for adjusting the second signal commensurate with the difference between the first and second signals.

11. A method of sensing the position of an end effector on a remotely manipulated robot arm relative to distinct features on a work surface having a known geometry, comprising the steps of:

mounting the arm so that the end effector is adjacent the work surface;

generating a video image of the end effector and the work surface in the vicinity of the end effector;

positioning the end effector at a known reference position relative to a reference feature on the work surface;

establishing a reference coordinate for the reference position of the end effector;

generating a digitized image of at least a portion of the video image;

calibrating the digitized image to the video image such that the reference coordinate of the end effector in the video image and the reference coordinate of the end effector in the digitized image are the same;

moving the arm relative to the work surface;

continually processing changes in the digitized image as the arm is moved, to continually track the coordinates of the path of movement of the end effector; and

displaying an index indicative of the actual coordinate of the end effector as the end effector is displaced from the reference position.

12. The method of claim 11 wherein the features have the same, known size and the step of calibrating includes the step of establishing a quantitative relationship between the size of the discrete features on the work surface as appear in the digitized image and the known size of the features.

13. The method of claim 12 wherein the step of processing the the changes includes,

continually capturing frames of digitized images of at least a portion of the video image that is generated as the arm is displaced,

scaling the digitized images in accordance with the quantitative relation established in the step of calibrating, and

comparing the changes in digitized images of at least two of the features contained in the frame resulting from changes in the video images accompanying displacement of the arm, with the known geometry and features of the work surface, to continually track the path of movement of the end effector.

14. The method of claim 13 wherein the step of comparing includes the steps of,

converting the digitized image of each of at least two features into a salient attribute that is characteristic of the feature, and

inferring from apparent movement of the salient attributes, the path of movement of the end effector.

15. The method of claim 14 wherein the features are tube ends and the salient attribute is the centroid of the tube end.

16. The method of claim 15 wherein,

the step of converting includes representing the centroid by a figurative window having a digitized image size at least equal to the digitized image size of a tube end and controlled to figuratively contain the centroid, and

the step of inferring includes,

figuratively establishing a boundary within the digitized frame image, and

relating changes in the relationship between each window and the boundary, to the known geometry of the work surface.

17. The method of claim 16 wherein the step of inferring further includes the step of eliminating a first window and substituting a second window in the step of relating, when the first window figuratively crosses the boundary.

18. The method of claim 15 further including the step of determining the centroids of the tube ends in the digitized image by computing and comparing the intensity profile of the tube image along two different axes.

19. The method of claim 11, further including the steps of,

selecting as a work surface, a tube sheet having as said features, tube ends of the same nominal diameter, and

storing a file containing data indicative of the geometry and dimensions of the work surface and the tube ends, and wherein,

the step of calibrating includes,

generating a digitized image of a reference tube, and

establishing a quantitative relation between the size of the digitized reference tube and the nominal diameter of the tube, and

the step of processing includes the steps of

continually capturing digitized frames of the video image and converting the digitized image of at least two tubes into salient attributes, and

from the quantitative relation, the data indicative of the geometry and dimensions of the work surface and tube ends, and the changes in the salient attributes accompanying displacement of the arm, tracking the path of movement of the end effector.

20. In a remote manipulator system including an articulated manipulator arm, an end effector on the manipulator arm, a manipulator control system for moving the arm to position the end effector sequentially in a desired relationship relative to discrete features on a work surface, and means mounted for movement with the arm for generating a video image of the work surface including a plurality of features as the arm is moved, the improvement comprising:

image sampling means for continually capturing a predetermined portion of the video image containing a plurality of features to produce a series of digitized image frames;

image conversion means coupled to the image sampling means, for extracting from each digitized image frame, a salient attribute of ach of at least two said features; and

tracking means coupled to the image conversion means, for inferring from changes in the salient attributes in the series of digitized frames, the path of movement of the arm relative to the work surface.

21. The system of claim 20 wherein,

the image conversion means extracts the salient attributes associated with the centroids of three adjacent features, and

the tracking means infers the changes in the positions of the three centroids relative to the centroids in the previously captured digitized image frames.

22. The system of claim 21, wherein the tracking means further includes,

means for generating a figurative boundary within the digitized image frames,

and wherein the changes in the positions of the three centroids are inferred relative to the boundary.

23. The system of claim 20, further including means coupled to the means for generating a video image and coupled to the tracking means, for displaying on the video image, an index indicative of the current position of the end effector relative to the work surface.