Robot system having image processing function

• US 7,177,459 B1
• Filed: 04/10/2000
• Issued: 02/13/2007
• Est. Priority Date: 04/08/1999
• Status: Expired due to Fees

• Current Assignee: Fanuc Ltd (Fanuc Corporation)
• Original Assignee: Fanuc Ltd (Fanuc Corporation)
• Inventors: Watanabe, Atsushi, Arimatsu, Taro
• Primary Examiner(s): Wu; Jingge
• Assistant Examiner(s): Strege; John B
• Application Number:
  US09/546,392
• Time in Patent Office: 2,500 Days
• Field of Search: 382151-154, 901/47
• US Class Current: 382/151
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/37553 Two cameras one for coarse scanning, other for fine scanning
• CPC Class Codes: G05B 2219/37555 Camera detects orientation, position workpiece, points of workpiece
• CPC Class Codes: G05B 2219/37567 3-D vision, stereo vision, with two cameras
• CPC Class Codes: G05B 2219/39393 Camera detects projected image, compare with reference image, position end effector
• CPC Class Codes: G05B 2219/40053 Pick 3-D object from pile of objects
• CPC Class Codes: G05B 2219/40613 Camera, laser scanner on end effector, hand eye manipulator, local

Abstract

A robot system having an image processing function capable of detecting position and/or posture of individual workpieces randomly arranged in a stack to determine posture, or posture and position of a robot operation suitable for the detected position and/or posture of the workpiece. Reference models are created from two-dimensional images of a reference workpiece captured in a plurality of directions by a first visual sensor and stored. Also, the relative positions/postures of the first visual sensor with respect to the workpiece at the respective image capturing, and relative position/posture of a second visual sensor to be situated with respect to the workpiece are stored. Matching processing between an image of a stack of workpieces captured by the camera and the reference models are performed and an image of a workpiece matched with one reference model is selected. A three-dimensional position/posture of the workpiece is determined from the image of the selected workpiece, the selected reference model and position/posture information associated with the reference model. The position/posture of the second visual sensor to be situated for measurement is determined based on the determined position/posture of the workpiece and the stored relative position/posture of the second visual sensor, and precise position/posture of the workpiece is measured by the second visual sensor at the determined position/posture of the second visual sensor. A picking operation for picking out a respective workpiece from a randomly arranged stack can be performed by a robot based on the measuring results of the second visual sensor.

Claims

1. A robot system having an image processing function for determining orientation, or orientation and position of a robot operation on one of a plurality of objects, the system comprising:

a robot;

a first image capturing device capturing image data of the plurality of objects containing respective images of the objects;

a memory storing reference models, each comprising an image of a reference object captured by said image capturing device in a different direction, and for each reference model storing information of the capturing direction of its associated image and information of an orientation of the robot with respect to the reference object, the information of the capturing direction representing a rotational posture of the reference object relative to the robot, said reference object being one of the plurality of objects or an object having a shape identical to that of one of the plurality of objects; and

a processor to perform matching on the image data containing images of the plurality of objects captured by said first image capturing device with each of said reference models successively to select one object having an image matched with one of said reference models, and to determine orientation, or orientation and position of the robot operation based on the image of the selected one object, based on said one reference model and the information of its associated capturing direction, and based on the information of the orientation of the robot operation with respect to the reference object that is associated with said one reference model.

2. A robot system having an image processing function according to claim 1, wherein said reference models are obtained from a part of the image data of the reference object.

3. A robot system having an image processing function according to claim 1, wherein said reference models are obtained by processing the image data of the reference object.

4. A robot system having an image processing function according to claim 1, wherein said first image capturing device comprises a camera for capturing two-dimensional image data.

5. A robot system having an image processing function according to claim 4, wherein said image data of the reference object are captured by said camera from a predetermined distance.

6. A robot system having an image processing function according to claim 1, further comprising:

a second image capturing device;

wherein

said robot situates said second image data capturing device to have said determined orientation or to have said determined orientation and said determined position with respect to the selected one object, and wherein

said processor processes second image data captured by said second image capturing device to detect position and/or rotational posture of the selected one object with respect to said second image data capturing device.

7. A robot system having an image processing function according to claim 1 further comprising:

a second image capturing device for obtaining three-dimensional position;

wherein

said robot situates said second image data capturing device to have said determined orientation or to have said determined orientation and said determined position with respect to the selected one object, so that said second image data capturing device is directed to a characterizing portion of the object; and

wherein

said processor detects three-dimensional position and/or posture of the selected one object based on three-dimensional position of said characterizing portion obtained by said second image capturing device.

8. A robot system having an image processing function according to claim 6, wherein said first image data capturing device is used as said second image data capturing device.

9. A robot system having an image processing function according to claim 6, wherein said second image capturing device comprises a three-dimensional visual sensor of spot-light scanning type capable of measuring distance between the sensor and an object.

10. A robot system having an image processing function according to claim 6, wherein said second image data capturing device comprises a structured-light unit for irradiating a structured light on the selected object and capturing an image of the object including the irradiated light on the object.

11. A robot system having an image processing function according to claim 7, wherein said robot operation is an operation of picking up the selected one object from the plurality of objects, some of which are overlapped with each other.

27. A robot system having an image processing function according to claim 7, wherein said first image data capturing device is used as said second image data capturing device.

28. A robot system having an image processing function according to claim 7, wherein said second image capturing device comprises a three-dimensional visual sensor of spot-light scanning type capable of measuring distance between the sensor and an object.

29. A robot system having an image processing function according to claim 7, wherein said second image data capturing device comprises a structured-light unit for irradiating a structured light on an object and capturing an image of the object including the irradiated light on the object.

12. A robot system having an image processing function for determining orientation, or orientation and position of a robot operation on one of a plurality of objects of plural kinds, where the orientation of the operation corresponds to a determined orientation of the one object, where the determined orientation is a rotational posture of the one object, the system comprising:

a robot;

a first image capturing device capturing image data of the plurality of objects containing respective images of the objects;

a memory storing reference models, each comprising images of each of different kinds of reference objects corresponding to images captured by said first image capturing device, and storing indicia of the kinds respectively associated with said reference models, and information of a different orientation of the robot with respect to each of the different images of the reference object of each kind, where the captured information of orientation comprises information of a rotational posture of the reference object relative to the robot, each of said reference objects being one of the kinds of the plurality of objects or having a shape identical thereto; and

a processor to perform matching on the image data containing images of the plurality of objects captured by said first image capturing device with each of said reference models successfully to select one object having an image matched with one of said kinds of the reference models, and to determine orientation, or orientation and position of the robot operation, the determining based on the image of the selected one object, based on said one reference model, based on the indicia of the kind associated with said one reference model and the information of the orientation of the robot operation with respect to the reference object associated with said one reference model of said one kind.

13. A robot system having an image processing function according to claim 12, wherein said reference models are obtained from a part of the image data of the reference object.

14. A robot system having an image processing function according to claim 12, wherein said reference models are obtained by processing the image data of the reference object.

15. A robot system having an image processing function according to claim 12, wherein said first image capturing device comprises a camera for capturing two-dimensional image data.

16. A robot system having an image processing function according to claim 15, wherein said image data of the reference object are captured by said camera from a predetermined distance.

17. A robot system having an image processing function according to claim 12, further comprising a second image capturing device, wherein

said robot situates said second image data capturing device to have said determined orientation or to have said determined orientation and said determined position with respect to the object, and wherein

said processor processes second image data captured by said second image capturing device to detect position and/or posture of the selected one object with respect to said second image data capturing device.

18. A robot system having an image processing function according to claim 12, further comprising:

a second image capturing device for obtaining three-dimensional position;

wherein

said robot situates said second image data capturing device to have said determined orientation or to have said determined orientation and said determined position with respect to the selected one object, so that said second image data capturing device is directed to a characterizing portion of the object; and

wherein

said processor detects three-dimensional position and/or posture of the selected one object based on three-dimensional position of said characterizing portion obtained by said second image capturing device.

19. A robot system having an image processing function according to claim 17, wherein said first image data capturing device is used as said second image data capturing device.

20. A robot system having an image processing function according to claim 17, wherein said second image capturing device comprises a three-dimensional visual sensor of spot-light scanning type capable of measuring distance between the sensor and an object.

21. A robot system having an image processing function according to claim 17, wherein said second image data capturing device comprises a structured-light unit for irradiating a structured light on the selected one object and capturing an image of the object including the irradiated light on the object.

22. A robot system having an image processing function according to claim 18, wherein said robot operation is an operation of picking up the selected one object from the plurality of objects, some of which are overlapped with each other.

30. A robot system having an image processing function according to claim 18, wherein said first image data capturing device is used as said second image data capturing device.

31. A robot system having an image processing function according to claim 18, wherein said second image capturing device comprises a three-dimensional visual sensor of spot-light scanning type capable of measuring distance between the sensor and an object.

32. A robot system having an image processing function according to claim 18, wherein said second image data capturing device comprises a structured-light unit for irradiating a structured light on an object and capturing an image of the object including the irradiated light on the object.

23. A method for automatically determining an arrangement of a workpiece relative to a robot, where the determined arrangement comprises at least rotational posture arrangement of the workpiece relative to the robot, the method comprising:

storing reference images corresponding to images of the workpiece or an object so shaped (workpiece/object) and reference arrangement information indicating arrangements of the robot and workpiece/object relative to each other when the images were captured, the reference arrangements comprising rotational arrangements of the workpiece relative to the robot;

from a known arrangement of the robot, capturing a working image of the workpiece among a plurality of randomly arranged workpieces with an imaging device;

finding one of the reference images that matches the workpiece in the working image; and

determining an arrangement of the robot relative to the workpiece based on information indicating the known arrangement of the robot, and based on the reference arrangement information corresponding to the found reference image, where the determined arrangement comprises rotational arrangement of the workpiece relative to the robot.

24. A method according to claim 23, wherein reference images and reference arrangement information is obtained for workpieces/objects of different shapes, and wherein the finding comprises first determining that a reference image of one of the different shapes matches the working image of the workpiece, and then finding one reference image of the shape that best matches the working image.

25. A method according to claim 23, wherein the robot is used to capture the reference images, and wherein the reference arrangement information represents arrangements of the robot when capturing the reference images.

26. A method according to claim 23, wherein a second imaging device is affixed to the robot and is used to determine additional arrangement information used to determine the known arrangement of the robot relative to the workpiece.