Robotic acquisition of objects by means including histogram techniques

• US 4,613,269 A
• Filed: 02/28/1984
• Issued: 09/23/1986
• Est. Priority Date: 02/28/1984
• Status: Expired due to Fees

• Current Assignee: Object Recognition Systems, Inc.
• Original Assignee: Object Recognition Systems, Inc.
• Inventors: Wilder, Joseph, Ray, Rajarshi
• Primary Examiner(s): Harkcom, Gary V.
• Application Number:
  US06/584,539
• Time in Patent Office: 938 Days
• Field of Search: 364/513, 364/478, 318/640, 318/568, 358/88-91, 358/100-101, 358/139, 358/903, 414/730, 414/732, 414/738, 901/14-18, 901/31, 901/34, 901/46-47
• US Class Current: 700/259
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/40053 Pick 3-D object from pile of objects
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20216 Image averaging
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 2201/06 Recognition of objects for industrial automation

Abstract

A robotic apparatus and method of acquiring jumbled objects or workpieces from a bin, and transferring them to a final site, or to an intermediate site for quality inspection. A video camera system produces a video signal representing a brightness image of the jumbled objects. An intermediate amplitude range of the video signal is selected, expanded, stored, and quantized into digital pixels. A computer enhances the image using histogram techniques, performs edge suppression, performs repeated shrinking with progressively-increased pixel thresholds until an area limit is reached, performs clustering of closely-spaced pixels, and commands movement of the robot gripper to a computed grip site on a recognized object. The gripper may limit its gripping pressure as a function of the deformation of the object gripped.

Claims

1. A robot system for acquiring randomly-arranged workpieces, comprising

a robot assembly including a hand, and a controller for controlling the movement of the hand to positions to successively grasp workpieces of given size and transfer them to a predetermined site,

a video imaging device to provide a video signal representing a brightness image of the randomly-arranged workpieces, and

a computer programmed to be responsive to the video signal from the imaging device and operative to modify the image, to compute a holdsite on a workpiece in the image, and to command the robot assembly to grasp the workpiece and transfer it to said predetermined site,

said video imaging device including a video range expander for selecting and amplifying a range of video signal amplitudes between a gray threshold level and a white clipping level,

said computer including program means to quantize the selected video signal having said range of amplitudes into digital pixels within a plurality of windows all having a size proportional to the size of the workpieces, said means to quantize the selected video signal including means to generate the same number of digital pixels of the image within the window regardless of the size of the window, and to enhance the quantized image by making all pixels having a brightness intensity less than a threshold intensity T equal to zero, and retaining the brightness intensities of all pixels above the threshold T, said threshold intensity T being computed by program from an intensity histogram so that T exceeds the intensity of the most numerous dark pixels by a predetermined fraction of the difference between the intensity of the most numerous dark pixels and the intensity of the most numerous bright pixels.

2. A system according to claim 1 wherein said computer includes program means to suppress the edges of the enhanced video image by reducing to zero all pixels whose brightness compared with the brightness of any adjacent pixel is greater than a predetermined ratio.

3. A system according to claim 2 wherein the brightness ratio of adjacent pixels is computed by determining the difference of the logarithms of the reflected intensities of the two pixels.

4. A system according to claim 2 wherein the brightness of each pixel is compared with the brightness of adjacent pixels along adjacent horizontal rows, and then the brightness of each pixel is compared with the brightness of adjacent pixels along adjacent vertical columns.

5. A system according to claim 1 wherein said computer includes program means to shrink the digitized image having an area A to an image having an area A_n by making each pixel equal to zero which has an intensity less than a predetermined initial threshold, or which is adjacent to a pixel having an intensity less than the predetermined threshold.

6. A system according to claim 5 wherein said computer includes program means to repeat said shrinking with progressively higher thresholds until the ratio of the area A to the area A_n exceeds a predetermined ratio.

7. A system according to claim 6 wherein said predetermined ratio of areas is computed during a training phase to be in the range from 1/2 the standard deviation, to three times the standard deviation greater than the average ratio of areas found in all of said plurality of windows by progressively shrinking the images in the plurality of windows until the ratio of each initial area to each final area is greater than a predetermined training ratio.

8. A system according to claim 6 wherein the repetition of said shrinking is stopped if the number of repetitions exceeds a predetermined number.

9. A system according to claim 6 wherein said computer includes program means to go back to the image arrived at after an earlier than last one of said progressively shrunk images.

10. A system according to claim 9 wherein said computer includes program means to form clusters of bright pixels by adding bright pixels in spaces less than a predetermined minimum distance between existing bright pixels.

11. A system according to claim 10 wherein said computer includes program means to identify said clusters having an area greater than a predetermined minimum area.

12. A system according to claim 11 wherein said computer includes program means to identify said large clusters which also have a length-width ratio greater than a predetermined minimum.

13. A system according to claim 12 wherein said computer includes program means to compute the location of a central point and the orientation about that point of a grasp site on a workpiece represented by an identified large elongated cluster of pixels, and to command the robot assembly to grasp the workpiece and transfer it to a predetermined site.

14. A system according to claim 13 wherein said computer includes program means to compute the center of area of the cluster of pixels to determine the location of the grasp site, and to compute a minimum dimension of the cluster of pixels to determine the orientation of the grasp site.

15. A system according to claim 1 wherein said hand is a parallel-jaw gripper provided with means to limit the workpiece-gripping pressure in accordance with the amount of compression of the gripped workpiece.

16. A system according to claim 1, and in addition, a second video imaging device to provide a video signal representing a workpiece which has been transferred to said predetermined site, and computer means programmed to examine the video signal from said second video imaging device and to compare the examined image with a least one predetermined reference image, and to command said robot assembly to transfer the workpiece to a site for workpieces corresponding with said reference image, and to another different site for workpieces not corresponding with said reference image.

17. A robot system for acquiring non-touching randomly-arranged workpieces, comprising

a robot assembly including a hand, and a controller for controlling the movement of the hand to positions to successively grasp workpieces of given size and transfer them to a predetermined site,

a video imaging device to provide a video signal representing a brightness image of the randomly-arranged workpieces, and

a computer programmed to be responsive to said video signal from the imaging device and operative to modify the image, to compute a holdsite on a workpiece in the image, and to command the robot assembly to grasp the workpiece and transfer it to said predetermined site,

said video imaging device including a video range expander for selecting and amplifying a range of video signal amplitudes between a gray threshold level and a white coupling level,

said computer including program means to quantize the selected video signal having said range of amplitudes into digital pixels within a plurality of windows all having a size proportioned to the size of the workpieces,

program means to enhance the quantized image by making all pixels having a brightness intensity less than a threshold T equal to zero, and retaining the brightness intensities of all pixels above the threshold T, wherein said threshold intensity T is computed from an intensity histogram by program means which selects the threshold intensity T at a value intermediate that of the most numerous dark pixels and that of the most numerous bright pixels, and

program means to determine whether the bright pixel area in a given window is greater than an area, computed during a training phase, which is in the range of 1/2 a standard deviation, to three standard deviations less than, the average of the areas found in all of said plurality of windows.

18. A system according to claim 17 wherein said computer includes program means to form clusters of bright pixels by adding bright pixels in spaces less than a predetermined minimum distance between existing bright pixels.

19. A system according to claim 18 wherein said computer includes program means to identify said clusters having an area greater than a predetermined minimum area.

20. A system according to claim 19 wherein said computer includes program means to identify said large clusters which also have a length-width ratio greater than a predetermined minimum.

21. A system according to claim 20 wherein said computer includes program means to compute the location and orientation of a grasp site on a workpiece represented by an identified large elongated cluster of pixels, and to command the robot assembly to grasp the workpiece and transfer it to a predetermined site.

22. A system according to claim 21 wherein said computer includes program means to compute the center of area of the cluster of pixels to determine the location of the grasp site, and to compute the axis of least moment of inertia of the cluster of pixels to determine the orientation of the grasp site.

23. The processing of a video image of randomly-arranged and touching workpiece of given size, comprising the steps of

selecting and amplifying a range of video signal amplitudes between a gray threshold level and a white clipping level,

quantizing the selected video signal, having said range of amplitudes, within a window having a size proportional to the size of the workpieces,

enhancing the digitized video image by making all pixels having a brightness intensity less than a threshold T equal to zero, where the threshold T is computed from an intensity histogram by means of a computer program which selects the threshold intensity T at a value intermediate that of the most numerous dark pixels and that of the most numerous bright pixels,

suppressing the edges in the enhanced video image by reducing to zero all pixels whose brightness compared with the brightness of any adjacent pixel is greater than a predetermined ratio R,

shrinking the edge-suppressed image having an area A to an image having an area A_n by making equal to zero each pixel which has an intensity less than a predetermined initial threshold, or which is adjacent to a pixel having an intensity less that the predetermined threshold,

repeating said shrinking with progressively higher thresholds until the ratio of the area A to the area A_n is larger than a predetermined limit ratio, or until a predetermined number of iterations is exceeded,

identifying clusters each consisting of adjacent bright pixel in a earlier than the last one of the progressively shrunk images,

merging clusters of bright pixels by adding bright pixels in spaces which are less than a predetermined minimum distance between existing bright pixels,

identifying merged clusters having areas greater than a predetermined minimum, and having a length-width ratio greater than a predetermined minimum, and

computing a grasp location and orientation of an identified cluster.

24. The processing of a video image of randomly-arranged non-touching workpieces of given size, comprising the steps of

selecting and amplifying a range of video signal amplitudes between a gray threshold level and a white clipping level,

quantizing the selected video signal, having said range of amplitudes, within a window having a size proportional to the size of the workpieces,

enhancing the digitized video image by making all pixels having a brightness intensity less than a threshold T equal to zero, where the threshold T is computed from an intensity histogram by means of a computer program which selects the threshold intensity T at a value intermediate that of the most numerous dark pixels and that of the most numerous bright pixels,

identifying clusters each consisting of adjacent bright pixels,

merging clusters of bright pixels adding bright pixels in spaces which are less than a predetermined minimum distance between existing bright pixels,

identifying merged clusters having areas greater than a predetermined minimum, and having a length-width ratio greater than a predetermined minimum, and

computing a grasp location and orientation of an identified cluster.

25. A robot system for acquiring randomly-arranged workpieces of given size, comprising

a robot assembly including a hand, and a controller for controlling the movement of the hand to positions to successively grasp workpieces and transfer them to a predetermined site,

a video imaging device to provide a video signal representing a brightness image of the randomly-arranged workpieces, and

a computer programmed to be responsive to the video signal from the imaging device and operative to modify the image, to compute a holdsite on a workpiece in the image, and to command the robot assembly to grasp the workpiece and transfer it to said predetermined site,

said video imaging device including a video range expander for selecting and amplifying a range of video signal amplitudes between a gray threshold level and a white clipping level,

said computer including program means to quantize the selected video signal having said range of amplitudes into digital pixels within a plurality of windows all having a size proportioned to the size of the workpieces,

program means to enhance the quantized image by making all pixels having a brightness intensity less than a threshold T equal to zero, and retaining the brightness intensities of all pixels above the threshold T,

program means to form clusters of bright pixels by adding bright pixels in spaces less than a predetermined minimum distance between existing bright pixels,

program means to identify said clusters having an area greater than a predetermined minimum area,

program means to identify said large clusters which also have a length-width ratio greater than a predetermined minimum,

program means to compute the location and orientation of a grasp site on a workpiece represented by an identified large elongated cluster of pixels, and to command the robot assembly to grasp the workpiece and transfer it to a predetermined site, and

program means to compute the center of area of the cluster of pixels to determine the location of the grasp site, and to compute the axis of least moment of inertia of the cluster of pixels to determine the orientation of the grasp site.

26. A system according to claim 25 wherein the computation of the center of area of pixels includes computing an initial center of area by finding the location of the pixel which occurs half way on a rasterscan through the cluster of bright pixels, and computing a corrected center of area by averaging the bright pixels on a small plurality of horizontal and vertical scans passing in the immediate vicinity of said initial center of area.

27. A system according to claim 26 and in addition the computation of a minimum dimension of the cluster of pixels by taking polar scans through said corrected center of area at angular increments, finding the scan length dimensions of the cluster of pixels under each polar scan line, smoothing the lengths with a moving window, and selecting a polar angle of the shortest scan line dimension for the needed angular orientation of the robot hand over the grasp site.

28. A system according to claim 27 wherein the robot assembly is commanded to grasp the workpiece only if said shortest scan line dimension is less than the maximum open dimension of the robot hand.

29. A system according to claim 28 wherein the robot assembly is commanded to grasp the workpiece only if the ratio of said shortest scan line dimension to the longest scan line dimension is less than a predetermined value.

30. A system according to claim 27 where the robot assembly is commanded to grasp the workpiece only if the endpoints of a plurality of scan line dimensions near said shortest scan line dimension define lines having at least a predetermined degree of parallelism.