Method for the selection of physical objects in a robot system

• US 9,050,719 B2
• Filed: 05/05/2011
• Issued: 06/09/2015
• Est. Priority Date: 06/24/2010
• Status: Active Grant

• Current Assignee: MP Zenrobotics OY
• Original Assignee: Zenrobotics Oy (Terex Corporation)
• Inventors: Valpola, Harri, Lukka, Tuomas
• Primary Examiner(s): Badii, Behrang
• Assistant Examiner(s): TESTARDI, DAVID A
• Application Number:
  US13/806,426
• Publication Number:
  US 20130151007A1
• Time in Patent Office: 1,496 Days
• Field of Search: 700/245, 700/258, 700/259
• US Class Current: 1/1
• CPC Class Codes: B25J 13/08 by means of sensing devices, e.g. viewing or touching devices
• CPC Class Codes: B25J 13/085 Force or torque sensors B25J13/082, B25J13/084 take precedence
• CPC Class Codes: B25J 9/00 Programme-controlled manipulators
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39528 Measuring, gripping force sensor build into hand
• CPC Class Codes: G05B 2219/40053 Pick 3-D object from pile of objects
• CPC Class Codes: G05B 2219/40571 Camera, vision combined with force sensor
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement

Abstract

A method in which an apparatus receives first sensor data from first sensors and determines a target position from the data, the target position may be a position in space or an orientation of a gripper in a robot arm First instructions are issued to the robot arm or the gripper in order to move a gripper to the target position. Force feedback sensor data is received from force feedback sensors associated with either the robot arm or the gripper or from the first sensors. A failure in carrying out the first instructions is determined. Second sensor data is received from the at least one first sensor. Successful gripping of an object is determined from the second sensor data.

Claims

1. A method, comprising:

receiving first sensor data from at least one first sensor, wherein the first sensor is a digital camera oriented to view an arena;

determining a target position from the first sensor data, said target position comprising at least one of a position in space and an orientation for a gripper in association with a robot arm;

starting to issue at least one first instruction to at least one of the robot arm and the gripper in order to move the gripper to the target position;

receiving feedback sensor data from at least one force feedback sensor, associated with at least one of the robot arm and the gripper, to determine a failure in carrying out the at least one first instruction;

receiving second sensor data from the at least one first sensor;

determining a successful gripping of an object from the second sensor data;

receiving verification sensor data from at least one second sensor, in response to the determining of the successful gripping;

issuing at least one second instruction to the robot arm in order to move the arm to a predetermined position to release the grip of the gripper;

registering the successful gripping and the first sensor data to a first computer program stored in a non-transitory computer readable medium;

registering an unsuccessful gripping and the first sensor data to the first computer program, in response to a determined failure in carrying out the at least one first instruction;

predicting a likelihood for the successful gripping of a next object from new first sensor data using the information registered in the first computer program; and

issuing at least one new first instruction to at least one of the robot arm and the gripper, to grip the next object, based on the predicted likelihood.

2. The method according to claim 1, the method further comprising:

issuing at least one third instruction to the robot arm to move the gripper to a verification position.

3. The method according to claim 2, wherein the verification position corresponds to a verification environment where the at least one second sensor is provided the second sensor data regarding the object without interfering objects.

4. The method according to claim 3, wherein the verification environment has a constant background, which does not change with time.

5. The method according to claim 2, the method further comprising:

determining a type of the object from the verification sensor data from the at least one second sensor.

6. The method according to claim 1, wherein the starting to issue the at least one first instruction further comprises:

providing a first coordinates instruction to the robot arm;

receiving first intermediate sensor data from the at least one first sensor; and

determining an actual position of the gripper in a sensor coordinate system using the first intermediate sensor data.

7. The method according to claim 6, wherein the starting to issue the at least one first instruction further comprises:

determining an error between the actual position of the gripper and the first coordinates instruction;

providing a second coordinates instruction to the robot arm;

receiving second intermediate sensor data from the at least one first sensor;

determining a second actual position of the gripper in the sensor coordinate system using the second intermediate sensor data; and

checking whether the error is smaller than a predetermined threshold.

8. The method according to claim 6, wherein the starting to issue the at least one first instruction further comprises:

registering connection information between coordinates of the robot arm and the position of the gripper as determined in the sensor coordinate system in a second computer program stored in the non-transitory computer readable medium.

9. The method according to claim 8, wherein the starting to issue the at least one first instruction further comprises:

constructing a conversion function between coordinates of the robot arm and coordinates in the sensor coordinates system, using the registered connection information.

10. The method according to claim 9, wherein the determining a target position from the first sensor data further comprises:

determining an initial target position in the sensor coordinate system; and

computing the target position from the initial target position using the conversion function.

11. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform:

receiving first sensor data from at least one first sensor, wherein the first sensor is a digital camera oriented to view an arena;

determining a target position from the first sensor data, said target position comprising at least one of a position in space and an orientation for a gripper in association with a robot arm;

starting to issue at least one first instruction to at least one of the robot arm and the gripper in order to move the gripper to the target position;

receiving force feedback sensor data from at least one force feedback sensor, associated with at least one of the robot arm and the gripper, to determine a failure in carrying out the at least one first instruction;

receiving second sensor data from the at least one first sensor;

determining a successful gripping of an object from the second sensor data;

receiving verification sensor data from at least one second sensor, in response to the determining of the successful gripping;

issuing at least one second instruction to the robot arm in order to move the arm to a predetermined position to release the grip of the gripper;

registering the successful gripping and the first sensor data to a first computer program stored in a non-transitory computer readable medium;

registering an unsuccessful gripping and the first sensor data to the first computer program, in response to a determined failure in carrying out the at least one first instruction;

predicting a likelihood for the successful gripping of a next object from new first sensor data using the information registered in the first computer program; and

issuing at least one new first instruction to at least one of the robot arm and the gripper, to grip the next object, based on the predicted likelihood.

12. An apparatus, comprising:

means for receiving first sensor data from at least one first sensor, wherein the first sensor is a digital camera oriented to view an arena;

means for determining a target position from the first sensor data, said target position comprising at least one of a position in space and an orientation for a gripper in association with a robot arm;

means for starting to issue at least one first instruction to at least one of the robot arm and the gripper in order to move the gripper to the target position;

means for receiving force feedback sensor data from at least one force feedback sensor, associated with at least one of the robot arm and the gripper, to determine a failure in carrying out the at least one first instruction;

means for receiving second sensor data from the at least one first sensor;

means for determining a successful gripping of an object from the second sensor data;

means for receiving verification sensor data from at least one second sensor, in response to the determining of the successful gripping;

means for issuing at least one second instruction to the robot arm in order to move the arm to a predetermined position to release the grip of the gripper;

means for registering the successful gripping and the first sensor data to a first computer program stored in a non-transitory computer readable medium;

means for registering an unsuccessful gripping and the first sensor data to the first computer program, in response to a determined failure in carrying out the at least one first instruction;

means for predicting a likelihood for the successful gripping of a next object from new first sensor data using the information registered in the first computer program; and

means for issuing at least one new first instruction to at least one of the robot arm and the gripper, to grip the next object, based on the predicted likelihood.

13. A non-transitory computer readable medium storing a computer program, the computer program comprising code for controlling a processor to execute a method comprising:

receiving first sensor data from at least one first sensor;

determining a target position from the first sensor data, said target position comprising at least one of a position in space and an orientation for a gripper in association with a robot arm;

starting to issue at least one first instruction to at least one of the robot arm and the gripper in order to move the gripper to the target position;

receiving force feedback sensor data from at least one of at least one force feedback sensor, associated with at least one of the robot arm and the gripper, and the at least one first sensor, to determine a failure in carrying out the at least one first instruction;

receiving second sensor data from the at least one first sensor;

determining a successful gripping of an object from the second sensor data;

receiving verification sensor data from at least one second sensor, in response to the determining of the successful gripping;

issuing at least one second instruction to the robot arm in order to move the arm to a predetermined position to release the grip of the gripper;

registering the successful gripping and the first sensor data to a first computer program stored in a non-transitory computer readable medium;

registering an unsuccessful gripping and the first sensor data to the first computer program, in response to a determined failure in carrying out the at least one first instruction;

predicting a likelihood for the successful gripping of a next object from new first sensor data using the information registered in the first computer program; and

issuing at least one new first instruction to at least one of the robot arm and the gripper, to grip the next object, based on the predicted likelihood.

14. The non-transitory computer readable medium according to claim 13, wherein said computer readable medium is a removable memory card, a holographic memory, a magnetic disk or an optical disk.