Method and device for visualizing computer-generated informations

• US 7,298,385 B2
• Filed: 02/10/2004
• Issued: 11/20/2007
• Est. Priority Date: 02/11/2003
• Status: Active Grant

• Current Assignee: KUKA Laboratories GmbH (Midea Group Co., Ltd)
• Original Assignee: KUKA Roboter GmbH (Midea Group Co., Ltd)
• Inventors: Kazi, Arif, Bischoff, Rainer, Kuepeli, Bahadir
• Primary Examiner(s): WANG, JIN CHENG
• Application Number:
  US10/775,918
• Publication Number:
  US 20040189631A1
• Time in Patent Office: 1,379 Days
• Field of Search: 345629-641, 345473-474, 700245-264, 700/166
• US Class Current: 345/633
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: G05B 2219/35422 Unit freely movable in space, detect its position, orientation by triangulation
• CPC Class Codes: G05B 2219/35506 Camera images overlayed with graphics, model
• CPC Class Codes: G05B 2219/36167 Use camera of handheld device, pda, pendant, head mounted display
• CPC Class Codes: G05B 2219/39443 Portable, adapted to handpalm, with joystick, function keys, display
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

To improve the operation of robots in installations reprogramming and emptying an operating sequence to be performed by them, the invention provides a method and a device for visualizing computer-assisted information in an image of the real environment on a viewing device, in which there is a determination of the position and orientation or pose of the image receiving device and that robot-specific informations corresponding to this determination are faded over the image of the real environment on the viewing device.

Claims

1. A method for superimposing computer-generated information into an image of the real environment detected by an image receiving unit located on a viewing device, the method comprising the steps of:

providing a robot;

providing a control unit;

providing the viewing device with the image receiving unit;

detecting an image of the real environment and detecting an image of the robot by the image receiving unit;

displaying said robot image and said real environment image on said viewing device;

obtaining a position and an orientation or pose of said robot based on said robot image and said real environment image detected via the image receiving unit;

obtaining robot-specific information from said robot via said control unit, said robot-specific information corresponding to information necessary for moving said robot;

superimposing the robot-specific information over the image of said robot on said viewing device, at least one robot-specific coordinate system being superimposed into said image of the robot and real environment; and

controlling said robot based on said robot-specific information superimposed over image of said robot such that movement of said image corresponds to movement of said robot.

2. The method according to claim 1, wherein said robot-specific image includes a hand flange-fixed coordinate system.

3. The method according to claim 1, wherein said robot-specific information includes robot axes.

4. The method according to claim 1, wherein an image of a control element is superimposed on said robot image, said control element being movable in at least two dimensions for robot manually programmed movement.

5. The method according to claim 4, wherein an image of the control unit in association and orientation of the robot hand flange is superimposed on said robot image.

6. The method according to claim 1, wherein at least one tool moved by a robot, preferably several robot elements are superimposed into said real environment.

7. The method according to claim 1, wherein said robot-specific information includes a robot path to be covered by said robot in connection with a working process, particularly that of the tool center point, optionally including additional information.

8. The method according to claim 1, wherein said robot-specific information includes path points, wherein the position of the path points in space is variable and in each case a robot path curve corresponding to the position of all the path points.

9. The method according to claim 1, wherein the image of a workpiece to be machined by a robot, optionally together with axes associated therewith is superimposed on said robot image.

10. The method according to claim 1, wherein for adapting a robot path to the position of a detected, real workpiece, a virtual image of the workpiece with a robot path adapted thereto is superimposed on said image of said real environment, so that by superimposing the virtual workpiece image with the image of the real workpiece it is possible to adapt the robot path to be performed to the position of the real workpiece.

11. The method according to claim 1, wherein a working area reachable by a robot and/or a permitted operating area is visualized on the viewing device.

12. The method according to claim 1, wherein movement corridors of a robot tool, robot hand and/or further robot elements are visualized on the viewing device.

13. The method according to claim 1, wherein permanent and/or instantaneous associations of at least one manual programmer of at least one robot are visualized.

14. The method according to claim 1, wherein the position and orientation of the display are detected by fixed markings in space.

15. The method according to claim 14, wherein the position and orientation of the viewing device are determined by radio.

16. The method according to claim 1, wherein the position and orientation of the viewing device are determined optically.

17. The method according to claim 1, wherein the robot-specific, computer-generated information is superimposed onto data spectacles to be worn by a user.

18. The method according to claim 1, wherein the robot-specific, computer-generated information is superimposed onto a transparent screen as the viewing device.

19. The method according to claim 1, wherein the robot-specific, computer-generated information is displayed together with optoelectronically recorded information on a graphicable screen as the viewing device.

20. The method according to claim 1, wherein the robot-specific, computer-generated information is displayed together with optoelectronically recorded information on a screen in the form of a graphicable screen of a manual programmer for a robot.

21. The method according to claim 1, wherein the robot-specific, computer-generated information is displayed together with optoelectronically recorded information on a graphicable screen as the viewing device connectable to a manual programmer for a robot.

22. The method according to claim 1, wherein said robot-specific information includes position of a tool center point frame, a tool contour, boundaries of a working space, axes of rotation of a plurality of robot articulations and travel speed of said robot.

23. A device for visualizing computer-generated information in an image of the real environment, the device comprising:

a robot;

an image receiving device;

a viewing device connected to said image receiving device, said image receiving device detecting an image of said robot and surrounding real environment of said robot, said viewing device displaying said robot image and said surrounding real environment;

a means for obtaining the position and orientation or pose of said robot based on said image of said robot and surrounding real environment of said robot detected via said image receiving device;

a control unit, said control unit receiving robot-specification information from said robot;

a means for superimposing said robot-specific information over said image of said robot and said surrounding real environment on said viewing device, said robot-specific information corresponding to information necessary for moving said robot, said control unit controlling said robot based on said robot-specific information superimposed on said image of said robot, whereby said robot is moved based on movement of said image of said robot.

24. The device according to claim 23, wherein said robot-specific information includes a hand flange-fixed coordinate system.

25. The device according to claim 23, wherein said robot-specific information includes robot axes.

26. The device according to claim 23, further comprising a display of the image of an operating unit of a robot manual programmer movable in at least two dimensions.

27. The device according to claim 26, further comprising a display of an image of the operating unit in association and orientation of the robot hand flange.

28. The device according to claim 23, further comprising a display of at least one tool moved by a robot and preferably several robot elements, in an operating environment of a robot.

29. The device according to claim 23, wherein said robot-specific information includes a robot path to be covered during an operating process and optionally with additional information for a robot, particularly the hand flange of a robot.

30. The device according to claim 23, wherein said robot-specific information includes path points with respect to a variability of the position of the path points in space and in each case a display of a position of a robot path curve corresponding to all the path points.

31. The device according to claim 23, further comprising a display of an image of a workpiece to be machined by a robot and optionally with axes associated therewith.

32. The device according to claim 23, wherein said robot-specific information includes a virtual image of a workpiece with a robot path adapted thereto for adapting the robot path to the position of a detected, real workpiece, so that by superimposing the virtual workpiece image with the image of the real workpiece it is possible to adapt the robot path to be performed to the position of the real workpiece.

33. The device according to claim 23, wherein said robot-specific information includes a display of an operating area reachable by a robot and/or a permitted operating area on the viewing device.

34. The device according to claim 23, further comprising a display of movement corridors of a robot tool, a robot hand and/or further robot elements on the viewing device.

35. The device according to claim 23, further comprising a display of permanent and/or instantaneous associations of at least one manual programmer with at least one robot.

36. The device according to claim 23, wherein said robot-specific information includes fixed markings in space for detecting the position and orientation of the display.

37. The device according to claim 36, wherein the markings are radio receivers, particularly transmitters.

38. The device according to claim 23, wherein the markings can be detected by an optical receiver.

39. The device according to claim 23, further comprising data spectacles to be worn by a user for displaying said robot-specific information and computer-generated information.

40. The device according to claim 23, further comprising a graphicable screen for displaying said robot-specific information and computer-generated information together with optoelectronically recorded information.

41. The device according to claim 23, further comprising a viewing device in the form of a graphicable screen of a manual programmer for a robot for displaying said robot-specific information and computer-generated information together with optoelectronically recorded information.

42. The device according to claim 23, further comprising a graphicable screen connectable to a manual programmer of a robot as a viewing device for displaying said robot-specific information and computer-generated information together with optoelectronically recorded information.

43. The device according to claim 23, wherein said robot-specific information includes position of a tool center point frame, a tool contour, boundaries of a working space, axes of rotation of a plurality of robot articulations and travel speed of said robot.

44. A method for superimposing computer-generated information, the method comprising:

providing a robot;

providing a control unit;

providing a viewing device;

providing an image receiving unit;

detecting an image of said robot and detecting an image of the real environment surrounding said robot via said image receiving unit;

displaying said image of said robot and said image of said real environment on said viewing device;

determining position of said robot based on said image of said robot and said image of said real environment;

obtaining robot-specific information from said robot via said control unit, said robot-specific information corresponding to one of a position of a tool center point frame, a tool contour, boundaries of a working space, axes of rotation of a plurality of robot articulations and travel speed of said robot;

superimposing said robot-specific information over said image of said robot and said image of said real environment; and

controlling said robot based on said robot-specific information superimposed over image of said robot such that movement of said image of said robot corresponds to movement of said robot.