Process and device for determining the position and the orientation of an image reception means

• US 7,818,091 B2
• Filed: 09/28/2004
• Issued: 10/19/2010
• Est. Priority Date: 10/01/2003
• Status: Expired due to Fees

• Current Assignee: KUKA Laboratories GmbH (Midea Group Co., Ltd)
• Original Assignee: KUKA Roboter GmbH (Midea Group Co., Ltd)
• Inventors: Kazi, Arif, Bischoff, Rainer
• Primary Examiner(s): Tran; Khoi
• Assistant Examiner(s): Peche; Jorge O
• Application Number:
  US10/952,313
• Publication Number:
  US 20050131582A1
• Time in Patent Office: 2,212 Days
• Field of Search: 700/254, 700/252, 700/259, 700/262, 700/263, 700/260, 700/264, 700/246, 700/247, 700/248, 318/568.1, 318/568.11, 901/94, 901/95, 901/86, 901/163, 702/94, 702/95, 702/86, 702/163, 396/54, 396/55, 396/233
• US Class Current: 700/259
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/36167 Use camera of handheld device, pda, pendant, head mounted display
• CPC Class Codes: G05B 2219/39045 Camera on end effector detects reference pattern
• CPC Class Codes: G05B 2219/39046 Compare image of plate on robot with reference, move till coincidence, camera
• CPC Class Codes: G05B 2219/39057 Hand eye calibration, eye, camera on hand, end effector
• CPC Class Codes: G05B 2219/39114 Hand eye cooperation, active camera on first arm follows movement of second arm
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/39451 Augmented reality for robot programming
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

A process and a device are provided for determining the pose as the entirety of the position and the orientation of an image reception device. The process is characterized in that the pose of the image reception device is determined with the use of at least one measuring device that is part of a robot. The device is characterized by a robot with an integrated measuring device that is part of the robot for determining the pose of the image reception device.

Claims

1. A process for determining the pose as the entirety of the position and the orientation of an image reception means, the process comprising:

providing an image reception means;

recording an image with said image reception means;

providing an automatic handling device, said automatic handling device comprising a measuring means;

determining the pose of the image reception means with said measuring means of said automatic handling device;

providing an augmented reality system, said augmented reality system comprising an image-generating component;

delivering said determined pose of said image reception means and said recorded image into said image-generating component of said augmented reality system;

superimposing a virtual image on said recorded image in said augmented reality system to form a superimposed image;

displaying said superimposed image via said augmented reality system;

the user selecting an object in said superimposed image;

changing traveling commands of said automatic handling device with said augmented reality system such that an orientation of said image reception means corresponds to a position of said selected object, wherein said selected object is arranged in a center position of said recorded image of said image reception means when said orientation of said image reception means corresponds to said position of said selected object;

manually moving said automatic handling device to view the selected object from another perspective when the viewing point of the selected object is not suitable;

maintaining the image reception means with the automatic handling device in a manually selected orientation when the viewing point is suitable; and

storing the traveling commands.

2. A process in accordance with claim 1, wherein the pose of the image reception means is determined via angle sensors located in axle drives of said automatic handling device, said automatic handling device being a robot.

3. A process in accordance with claim 1, wherein said measuring means obtains pose data, said measuring means determining the pose of said image reception means, said pose data being assigned to the images recorded in a particular pose by said image reception means.

4. A process in accordance with claim 1, wherein said augmented reality system inserts virtual information of said automatic handling device into images recorded by the image reception means.

5. A process in accordance with claim 1, wherein an optical display is used as the image reception means.

6. A process in accordance with claim 1, wherein a camera is used as the image reception means.

7. A process in accordance with claim 6, wherein the camera is used at said automatic handling device, which is programmed especially to perform tasks.

8. A process in accordance with claim 6, wherein a camera is used on another automatic handling device located adjacent to said automatic handling device programmed to perform tasks.

9. A process in accordance with claim 1, wherein said measuring means is connected to said automatic handling device.

10. A process in accordance with claim 1, wherein the pose of said automatic handling device in the space is determined by means of at least one mark arranged in the space as well as at least one said measuring means arranged at a robot, such as a camera or a sensor.

11. A process in accordance with claim 1, wherein the pose of said automatic handling device in the space is determined by means of at least one mark arranged at said automatic handling device as well as at least one said measuring means arranged independently from said automatic handling device, such as a camera or at least one sensor.

12. A process in accordance with claim 1, wherein the pose of said automatic handling device relative to another automatic handling device is determined by means of a mark arranged on said automatic handling device and said measuring means, such as a camera or a sensor, which is arranged at said another automatic handling device.

13. A process in accordance with claim 1, wherein combining said recorded image with said virtual image to form said combined image includes overlaying said virtual image on said recorded image.

14. A device for determining a pose as the entirety of a position and an orientation of an image reception means, the device comprising:

an automatic handling device with a measuring means integrated with said automatic handling device for determining the pose of the image reception means, said image reception means recording an image;

an augmented reality system comprising an image-generating component, said image-generating component receiving said determined pose of said image reception means and said recorded image, wherein a virtual image of said augmented reality system is superimposed on said recorded image is to form a superimposed image, said augmented reality system generating an image reception means orientation signal when a position of a selected object in said superimposed image is changed in said augmented reality system, said image reception means receiving said image reception means orientation signal such that said image reception means is moved to a selected object position, said selected object position corresponding to said position of said selected object in said superimposed image, said selected object being arranged opposite a center position of said image reception means when said image reception means is in said selected object position; and

a control for manually moving said automatic handling device to view the selected object from another perspective when the viewing point of the selected object is not suitable, for maintaining the image reception means with the automatic handling device in a manually selected orientation when the viewing point is suitable, and for storing traveling commands of said automatic handling device.

15. A device in accordance with claim 14, wherein an assigning means is provided for assigning pose data, which determine the pose of an image reception means and are obtained by said measuring means of said automatic handling device, with the particular images recorded in a particular pose by the image reception means.

16. A device in accordance with claim 14, wherein a means for inserting specific virtual information is provided for inserting virtual information into images recorded by the image reception means.

17. A device in accordance with claim 14, wherein said measuring means of said automatic handling device has angle sensors integrated in robot axles of said automatic handling device.

18. A device in accordance with claim 14, wherein said image reception means is an optical display.

19. A device in accordance with claim 18, wherein the camera is arranged on said automatic handling device, which is programmed especially to perform a task.

20. A device in accordance with claim 14, wherein said image reception means is a camera.

21. A device in accordance with claim 14, wherein a measuring means for determining the site and the position of a said image reception means arranged separately from said automatic handling device is arranged on said automatic handling device.

22. A device in accordance with claim 14, further comprising fixed marks in space as well as on said measuring means arranged on said automatic handling device, for determining the pose of said automatic handling device relative to said marks.

23. A device in accordance with claim 14, wherein at least one said mark is arranged on said automatic handling device as well as at least one said measuring means, which is arranged independently from said automatic handling device, such as a camera or at least one sensor, for receiving the pose of the said automatic handling device in the space.

24. A device in accordance with claim 14, wherein a mark is arranged on said automatic handling device as well as said measuring means, which is arranged on another said automatic handling device, such as a camera or a sensor, for determining the relative positions of the said two automatic handling devices.

25. A process for determining the orientation and position of an image reception means, the process comprising:

providing an image reception means;

recording an image with said image reception means;

providing a robot, said robot comprising a measuring means;

determining a pose of the image reception means with said measuring means of said robot;

providing an augmented reality system, said augmented reality system comprising an image-generating component;

delivering said determined pose of said image reception means and said recorded image into said image-generating component of said augmented reality system;

superimposing a virtual image on said recorded image of said augmented reality system to form a superimposed image;

displaying said superimposed image via said augmented reality system;

the user selecting at least one object in said superimposed image;

changing an orientation of said image reception means such that said orientation of said image reception means corresponds to a position of said selected object in said superimposed image, said selected object being arranged in a center position of said recorded image;

manually moving said automatic handling device to view the selected object from another perspective when the viewing point of the selected object is not suitable;

maintaining the image reception means with the automatic handling device in a manually selected orientation when the viewing point is suitable; and

storing traveling commands of said automatic handling device.