Method and a system for programming an industrial robot

• US 20050256611A1
• Filed: 11/24/2004
• Published: 11/17/2005
• Est. Priority Date: 11/24/2003
• Status: Active Grant

• Current Assignee: ABB Schweiz AG (ABB Limited)
• Original Assignee: ABB Research Ltd. (ABB Ltd.)
• Inventors: Skourup, Charlotte, Pretlove, John, Pettersen, Thomas
• Granted Patent: US 7,353,082 B2
• Time in Patent Office: Days
• US Class Current: 700/264
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: G05B 19/42 Recording and playback systems, i.e. in which the programme is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine
• CPC Class Codes: G05B 2219/35444 Gesture interface, controlled machine observes operator, executes commands
• CPC Class Codes: G05B 2219/35482 Eyephone, head-mounted 2-D or 3-D display, also voice and other control
• CPC Class Codes: G05B 2219/36167 Use camera of handheld device, pda, pendant, head mounted display
• CPC Class Codes: G05B 2219/36492 Record position and orientation, posture of probe, tool
• CPC Class Codes: G05B 2219/36495 Recording position and other parameters, current, tool diameter, voltage
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/39451 Augmented reality for robot programming

Abstract

A method and a system for use in connection with off-line programming of an industrial robot. The robot is taught a path having a number of waypoints located on or in the vicinity of at least one object to be processed by the robot. The system includes a tracking system unit adapted to provide information about the position of a part of the body of an operator pointing by the part at points on or in the vicinity of the object, a visual feed-back unit generating a visual feed-back to the operator of the position of the point being presently pointed at by the part of the body, in relation to the object, and a storage unit adapted for storing the position of the part of the body as a waypoint upon receiving a recording command.

Claims

1. A method in connection with off-line programming of an industrial robot, comprising teaching the robot a path having a number of waypoints located on or in the vicinity of at least one object to be processed by the robot, the method comprising:

obtaining information about the position of a part of a body of an operator pointing by said part at a point on or in the vicinity of the object,

determining the position of the point being presently pointed at by said part of the body, in relation to the object, based upon said obtained information,

generating a visual feed-back to the operator of the position of the point being presently pointed at by said part of the body in relation to the object,

storing the position of said point being presently pointed at by said part of the body as a waypoint upon receiving a recording command, and

generating robot programming code based upon the stored waypoint.

2. The method according to claim 1, wherein said generation of a visual feed-back to the operator comprises:

generating a graphical representation of the point being presently pointed at by said part of the body of the operator, and

displaying a view comprising the object and said graphical representation projected on the object.

3. The method according to claim 2, wherein the method further comprises:

obtaining an image of the object,

registering the generated graphical representation of the point to the image of the object to provide a composite augmented reality image, and

displaying a view showing the object and said graphical representation projected on the image of the object.

4. The method according to claim 2, wherein said displaying step further comprises generating a graphical representation of the stored waypoint and displaying a view comprising the object and said graphical representation projected on the object.

5. The method according to claim 2, further comprising:

generating a graphical representation of the path between the latest stored waypoint and the point being presently pointed at by said part of the body of the operator and displaying a view comprising the object and said graphical representation projected on the object.

6. The method according to claim 2, further comprising:

generating a graphical representation of the path between at least some of the stored waypoints and displaying a view comprising the object and said graphical representation projected on the object.

7. The method according to claim 1, wherein obtaining information about the position of said part of the body comprises:

obtaining an image of the object and of said part of the body, and

determining the position of the point being presently pointed at by said part of the body in relation to the object based upon said image.

8. The method according to claim 1, further comprising:

obtaining information about the orientation of said part of the body,

storing the orientation and position of said part of the body as a waypoint upon receiving the recording command, and

generating said robot programming code based upon the stored waypoint.

9. The method according to claim 1, wherein said part of the body is a finger, and/or a hand, and/or an arm.

10. The method according to claim 1, further comprising:

storing at least one predefined gesture representing a specific command,

obtaining information about the configuration and/or movements of at least a part of the body of the operator,

recognizing a gesture based on the obtained information about the body, and

interpreting the command by comparing the recognized gesture with said predefined gesture representing the command.

11. The method according to claim 10, wherein said obtained information is hand motion gestures.

12. The method according to claim 10 or 11, wherein said predefined gesture represents the recording command, and the recording command is identified by comparing the recognized gesture with said predetermined gesture representing the recording command.

13. The method according to claim 10, further comprising:

storing a plurality of predefined gestures, each representing a specific command,

recognizing gestures based on the obtained information about the body,

interpreting the recognized gestures by comparing a recognized gesture with the stored predefined gestures, and

executing the command represented by the recognized gesture.

14. The method according to claim 1, further comprising:

storing a plurality of predefined gestures, each representing a specific piece of process related information or an process event,

obtaining information about the configuration or movements of at least a part of the body of the operator,

recognizing gestures based on the obtained information about said part of the body,

interpreting the recognized gestures by comparing a recognized gesture with the stored predefined gestures, and

generating robot programming code based upon the process related information or process event represented by the recognized gesture.

15. A computer program directly loadable into the internal memory of a computer or a processor, comprising:

software code portions for performing the steps of claim 1, when said program is run on a computer.

16. A computer readable medium having a program recorded thereon, where the program is to make a computer perform the steps of claim 1, when said program is run on the computer.

17. A system for use in connection with off-line programming of an industrial robot, the programming comprises teaching the robot a path having a number of waypoints located on or in the vicinity of at least one object to be processed by the robot, the system comprising:

a tracking system unit adapted for providing information about the position of a part of the body of an operator pointing by said part at points on or in the vicinity of the object,

a visual feed-back unit generating a visual feed-back to the operator of the position of the point being presently pointed at by said part of the body, in relation to the object, and

a storage unit adapted for storing the position of said part of the body as a waypoint upon receiving a recording command.

18. The system according to claim 17, further comprising:

a graphical generator operative to generate a graphical representation of the point being presently pointed out by said part of the body, and

a display member operative to display a view comprising the object and said graphical representation projected on the object.

19. The system according to claim 18, further comprising:

a camera adapted for delivering an image of the object, and

means for registering the generated graphical representation of the point to the image of the object, to provide a composite augmented reality image

wherein the display member is adapted for displaying a view comprising the object and said graphical representation projected on the image of the object.

20. The system according to claim 18, wherein the graphical generator is adapted to generate a graphical representation of the stored waypoint.

21. The system according to claim 18, wherein the graphical generator is adapted to generate a graphical representation of the path between the latest stored waypoint and the point presently being pointed out.

22. The system according to claim 17, further comprising:

a camera adapted to deliver an image of at least said part of the body,

wherein the pose determining unit is adapted to determine the position of the part of the body in relation to the object based upon said image of the part of the body.

23. The system according to claim 17, wherein said part of the body is a finger, and/or a hand, and/or an arm.

24. The system according to claim 17, further comprising:

a robot programming-code generator operative to generate robot-programming code based upon the stored waypoint.

25. The system according to claim 17, wherein said tracking system unit is adapted for providing information about the configuration or movements of at least a part of the body of the operator, and the system further comprising:

a command storage unit adapted for storing predefined gestures representing specific commands,

a gesture recognition unit operative to recognize gestures based on the provided information about the body, and

a gesture interpretation unit operative to interpret the recognized gestures by comparing the recognized gestures with said predefined gestures and providing commands based on the interpretation.

26. The system according to claim 17, further comprising:

a process information storage unit adapted to store a plurality of predefined gestures, each representing a specific piece of process related information or a process event,

a gesture recognition unit operative to recognize gestures based on the provided information about the body,

a gesture interpretation unit operative to interpret the recognized gestures by comparing a recognized gesture with said predefined gestures and providing the process related information or the process event based on the interpretation.

27. The system according to claim 17, wherein said tracking system unit is adapted to determine the orientation of said part of the body in relation to the object, based upon said provided information, and wherein said storage unit is adapted to store the position and orientation of said part of the body as a waypoint upon receiving the recording command.

28. Use of a system according to claim 17 for programming one or more robots and/or automation applications to carry out an operation comprising any from the list of:

welding, riveting, de-burring, fettling, grinding, coating, painting, dry spraying, gluing, folding plate, bending plate, hemming plate, gripping an object, manipulating an object, stacking, pick and place.

29. Use of a system according to claim 17 for programming a mobile robot in an industrial installation or commercial installation or health care facility or place of work.