Method and system for highly precisely positioning at least one object in an end position in space

• US 8,798,794 B2
• Filed: 05/26/2010
• Issued: 08/05/2014
• Est. Priority Date: 05/27/2009
• Status: Active Grant

• Current Assignee: Leica Geosystems AG (Hexagon AB)
• Original Assignee: Leica Geosystems AG (Hexagon AB)
• Inventors: Walser, Bernd, Metzler, Bernhard, Aebischer, Beat, Siercks, Knut, Pettersson, Bo
• Primary Examiner(s): Black, Thomas
• Assistant Examiner(s): Nelson, Sara
• Application Number:
  US13/322,123
• Publication Number:
  US 20120072021A1
• Time in Patent Office: 1,532 Days
• Field of Search: 700/245, 700/250, 700/252, 700/253, 700/254, 700/259, 700/262, 901/2, 901/6, 901/8, 901/9, 901/11, 901 14- 18, 901/31, 901/47, 382/103, 382/106, 382/151, 382/153, 382/154
• US Class Current: 700/254
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 19/402 characterised by control arrangements for positioning, e.g. centring a tool relative to a hole in the workpiece, additional detection means to correct position G05B19/19 takes precedence
• CPC Class Codes: G05B 2219/36412 Fine, autonomous movement of end effector by using camera
• CPC Class Codes: G05B 2219/37555 Camera detects orientation, position workpiece, points of workpiece
• CPC Class Codes: G05B 2219/37567 3-D vision, stereo vision, with two cameras
• CPC Class Codes: G05B 2219/39387 Reflex control, follow movement, track face, work, hand, visual servoing
• CPC Class Codes: G05B 2219/39391 Visual servoing, track end effector with camera image feedback
• CPC Class Codes: G05B 2219/40555 Orientation and distance
• CPC Class Codes: G05B 2219/40604 Two camera, global vision camera, end effector neighbourhood vision camera
• CPC Class Codes: Y10S 901/08 Robot
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/14 Arm movement, spatial
• CPC Class Codes: Y10S 901/31 Gripping jaw
• CPC Class Codes: Y10S 901/47 Optical

Abstract

An object is highly precisely moved by an industrial robot to an end position by the following steps, which are repeated until the end position is reached within a specified tolerance: Recording a three-dimensional image by means of a 3-D image recording device. Determining the present position of the object in the spatial coordinate system from the position of the 3-D image recording device the angular orientation of the 3-D image recording device detected by an angle measuring unit, the three-dimensional image, and the knowledge of features on the object. Calculating the position difference between the present position of the object and the end position. Calculating a new target position of the industrial robot while taking into consideration the compensation value from the present position of the industrial robot and a value linked to the position difference. Moving the industrial robot to the new target position.

Claims

1. A method for highly precisely positioning at least one object in an end position in space by means of an industrial robot, comprising:

a first industrial robot, which is adjustable into predeterminable positions; and

an optical 3D image recording device, which;

is calibrated in a three-dimensional spatial coordinate system and is positioned in a position with a known alignment;

is designed for electronically recording three-dimensional images within a specific field of view, wherein the three-dimensional images are in each case composed of a multiplicity of pixels which are each assigned an item of depth information;

has a drive unit for the alignment, and which brings about adjustment of the field of view, of the 3D image recording device; and

has an angle measuring unit calibrated in the spatial coordinate system and serving for highly precisely detecting the angular alignment of the 3D image recording device, such that it is possible to determine the field of view in the spatial coordinate system,

wherein;

the position of the 3D image recording device in the spatial coordinate system is determined by aiming at stationary target marks by means of the 3D image recording device;

a first object, which has known optically detectable first features, is gripped and held by the first industrial robot within a gripping tolerance; and

a first compensation variable, correcting the gripping tolerance, for the first industrial robot is determined such that the first object is adjustable in a compensated fashion in the spatial coordinate system by predetermining a position of the first industrial robot, wherein the first compensation variable is determined by the steps of;

aligning the field of view of the 3D image recording device by means of the drive unit with at least one portion of the first features of the first object held in a first compensation position of the first industrial robot;

recording at least one first three-dimensional image;

determining the position of the first object in the spatial coordinate system in the first compensation position of the first industrial robot from the position of the 3D image recording device, the angular alignment of the 3D image recording device, said angular alignment being detected by the angle measuring unit, the first three-dimensional image, and the knowledge of the first features on the first object; and

determining the first compensation variable by utilizing the first compensation position of the first industrial robot, and at least the determined position of the first object in the first compensation position of the first industrial robot; and

the first object is highly precisely adjusted to a first end position by the following steps, which are repeated until the first end position is reached in a predetermined tolerance;

recording at least one further first three-dimensional image;

determining the present position of the first object in the spatial coordinate system from the position of the 3D image recording device, the angular alignment of the 3D image recording device, said angular alignment being detected by the angle measuring unit, the further first three-dimensional image, and the knowledge of the first features on the first object;

calculating the position difference between the present position of the first object and the first end position; and

calculating a new desired position of the first industrial robot taking account of the first compensation variable from the present position of the first industrial robot, and a variable which is linked to the position difference and is formed by the position difference multiplied by a factor of less than or equal to 1, and adjusting the first industrial robot to the new desired position.

2. The method as claimed in claim 1, wherein the stationary target marks and the known optically detectable first features of the first object are spaced apart in such a way that aiming at the stationary target marks and recording the at least one first three-dimensional image are effected with different alignments of the field of view of the 3D image recording device.

3. The method as claimed in claim 1, wherein after gripping the first object, the first industrial robot is adjusted to the first compensation position of the first industrial robot for determining the first compensation variable.

4. The method as claimed in claim 1, wherein before the steps which are repeated until the first end position is reached in a predetermined tolerance:

the first industrial robot is adjusted taking account of the first compensation variable from the first compensation position to a position in which the first object is positioned in a first approximation position near the first end position; and

the field of view of the 3D image recording device is aligned by means of the drive unit with at least one portion of the first features of the first object positioned in the first approximation position.

5. The method as claimed in claim 1, wherein before highly precisely adjusting the first object to the first end position, a second object is gripped by a second industrial robot or manually and is positioned into an object mount in a second end position in the spatial coordinate system.

6. The method as claimed in claim 5, wherein the second object has known optically detectable second features, and, after positioning the second object in the object mount, the second end position of the second object in the spatial coordinate system is determined by the steps of:

aligning the field of view of the 3D image recording device by means of the drive unit with at least one portion of the second features of the second object;

recording at least one second three-dimensional image; and

determining the second end position of the second object in the spatial coordinate system from the position of the 3D image recording device, the angular alignment of the 3D image recording device, said angular alignment being detected by the angle measuring unit, the second three-dimensional image, and the knowledge of the second features on the second object,

wherein the first end position for the first object is calculated from the second end position of the second object and a predetermined relative position between the first object and the second object.

7. The method as claimed in claim 1, wherein, before highly precisely adjusting the first object to the first end position:

a second object, which has known optically detectable second features, is gripped and held by a second industrial robot within a gripping tolerance;

the second industrial robot is adjusted to an end position of the second industrial robot, in which position the second object is situated in a second end position; and

the second end position of the second object in the spatial coordinate system is determined by the steps of;

aligning the field of view of the 3D image recording device by means of the drive unit with at least one portion of the second features of the second object;

recording at least one second three-dimensional image; and

determining the second end position of the second object in the spatial coordinate system from the position of the 3D image recording device, the angular alignment of the 3D image recording device, said angular alignment being detected by the angle measuring unit, the second three-dimensional image, and the knowledge of the second features on the second object,

wherein the first end position for the first object is calculated from the second end position of the second object and a predetermined relative position between the first object and the second object.

8. The method as claimed in claim 1, wherein, before highly precisely adjusting the first object to the first end position:

a second object, which has known optically detectable second features, is gripped and held by a second industrial robot within a gripping tolerance;

a second compensation variable, correcting the gripping tolerance, for the second industrial robot is determined such that the second object is adjustable in a compensated fashion in the spatial coordinate system by predetermining a position of the second industrial robot, wherein the second compensation variable is determined by the steps of;

aligning the field of view of the 3D image recording device by means of the drive unit with at least one portion of the second features of the second object held in a second compensation position of the second industrial robot;

recording at least one second three-dimensional image;

determining the position of the second object in the spatial coordinate system in the second compensation position of the second industrial robot from the position of the 3D image recording device, the angular alignment of the 3D image recording device, said angular alignment being detected by the angle measuring unit, the second three-dimensional image, and the knowledge of the second features on the second object; and

determining the second compensation variable by utilizing the second compensation position of the second industrial robot, and at least the determined position of the second object in the second compensation position of the second industrial robot; and

the second object is highly precisely adjusted to a second end position by the following steps, which are repeated until the second end position is reached in a predetermined tolerance;

recording at least one further second three-dimensional image,

determining the present position of the second object in the spatial coordinate system from the position of the 3D image recording device;

the angular alignment of the 3D image recording device, said angular alignment being detected by the angle measuring unit, the further second three-dimensional image, and the knowledge of the second features on the second object;

calculating the position difference between the present position of the second object and the second end position;

calculating a new desired position of the second industrial robot taking account of the second compensation variable from the present position of the second industrial robot, and a variable which is linked to the position difference and is formed, in particular, by the position difference multiplied by a factor of less than or equal to 1; and

adjusting the second industrial robot to the new desired position.

9. The method as claimed in claim 8, wherein after gripping the second object the second industrial robot is adjusted to the second compensation position of the second industrial robot for determining the second compensation variable.

10. The method as claimed in claim 8, wherein before the steps which are repeated until the second end position is reached in a predetermined tolerance:

the second industrial robot is adjusted taking account of the second compensation variable from the second compensation position to a position in which the second object is positioned in a second approximation position near the second end position; and

the field of view of the 3D image recording device is aligned by means of the drive unit with at least one portion of the second features of the second object positioned in the second approximation position.

11. The method as claimed in claim 1, wherein:

a third object embodied as a processing tool is held by a third industrial robot within a holding tolerance;

the processing tool or a part coupled thereto of the third industrial robot has known optically detectable third features; and

a third compensation variable, correcting the holding tolerance, for the third industrial robot is determined such that the processing tool is adjustable in a compensated fashion in the spatial coordinate system by predetermining a position of the third industrial robot, wherein the third compensation variable is determined by the steps of;

aligning the field of view of the 3D image recording device by means of the drive unit with at least one portion of the third features, the processing tool being held in a third compensation position of the third industrial robot;

recording at least one third three-dimensional image;

determining the position of the processing tool in the spatial coordinate system in the third compensation position of the third industrial robot from the position of the 3D image recording device, the angular alignment of the 3D image recording device, said angular alignment being detected by the angle measuring unit, the third three-dimensional image, and the knowledge of the third features; and

determining the third compensation variable by utilizing the third compensation position of the third industrial robot, and at least the determined position of the processing tool in the third compensation position of the third industrial robot.

12. The method as claimed in claim 11, wherein the processing tool is highly precisely adjusted to a third end position by the following steps, which are repeated until the third end position is reached in a predetermined tolerance:

recording at least one further third three-dimensional image;

determining the present position of the processing tool in the spatial coordinate system from the position of the 3D image recording device, the angular alignment of the 3D image recording device, said angular alignment being detected by the angle measuring unit, the further third three-dimensional image, and the knowledge of the third features;

calculating the position difference between the present position of the third object and the third end position;

calculating a new desired position of the third industrial robot taking account of the third compensation variable from the present position of the third industrial robot, and a variable which is linked to the position difference and is formed, in particular, by the position difference multiplied by a factor of less than or equal to 1; and

adjusting the third industrial robot to the new desired position.

13. The method as claimed in claim 1, wherein:

the features are substantially known from a model which can be processed by an electronic data processing device,

by means of image processing executed on an electronic data processing device, the features in the model and/or in the image recordings are identified and the features from the model and the features from the image recordings are assigned to one another; and

the position of the object in the spatial coordinate system is determined from the detected position of the recorded features in the spatial coordinate system and the assigned features.

14. The method as claimed in claim 13, wherein:

the relative position of the recorded features with respect to one another is determined and is compared with the relative position of the features substantially known from the model with respect to one another.

15. The method as claimed in claim 13, wherein:

the relative position of the recorded features with respect to one another is determined and is compared with the relative position of the features substantially known from the model with respect to one another;

a fault message is output when a deviation between the relative position of the recorded features and the relative position of the features substantially known from the model is exceeded;

in the case of the fault message, the object is exchanged for a new object, and

the model is adapted to the detected object and the adapted model determines the end position in the spatial coordinate system.

16. The method as claimed claim 1, wherein the 3D image recording device is embodied in such a way that:

at least one first camera can be aligned by means of the drive unit about in each case a horizontal tilting axis with respect to the spatial coordinate system and a vertical axis; and

the angle measuring unit in each case detects the horizontal angular alignment about the vertical axis and the vertical angular alignment about the tilting axis in the spatial coordinate system.

17. The method as claimed claim 1, wherein the 3D image recording device is embodied in such a way that:

at least one first camera can be aligned by means of the drive unit about in each case a horizontal tilting axis with respect to the spatial coordinate system and a vertical axis;

the angle measuring unit in each case detects the horizontal angular alignment about the vertical axis and the vertical angular alignment about the tilting axis in the spatial coordinate system; and

the horizontal tilting axis and the vertical axis substantially intersect.

18. The method as claimed in claim 17, wherein the first camera is embodied as an RIM camera for directly recording the three-dimensional image composed of a multiplicity of pixels each having assigned depth information.

19. The method as claimed in claim 17, wherein the 3D image recording device has an electro-optical distance measuring device and the three-dimensional image is generated by detecting the items of depth information by means of at least one optical depth measurement on sections of the object which correspond to the pixels by means of the electro-optical distance measuring device, and assigning the items of depth information to the multiplicity of pixels.

20. The method as claimed in claim 17, wherein:

the 3D image recording device has an electro-optical distance measuring device and the three-dimensional image is generated by detecting the items of depth information by means of at least one optical depth measurement on sections of the object which correspond to the pixels by means of the electro-optical distance measuring device, and assigning the items of depth information to the multiplicity of pixels; and

the electro-optical distance measuring device;

is embodied as a point scanner and the items of depth information are detected by serial scanning by means of the point scanner, wherein at least one point-like measurement beam optically scans point by point an area of the object which corresponds to the field of view;

or

is embodied as a line scanner and the items of depth information are detected by parallel scanning by means of the line scanner, wherein at least one line-like measurement beam optically scans line by line an area of the object which corresponds to the field of view;

or

is embodied as an area depth measuring device and the items of depth information are detected by fully parallel scanning by means of the area depth measuring device, wherein at least one areal measurement beam, formed in particular by a bundle composed of a multiplicity of measurement beams, optically scans area by area an area of the object which corresponds to the field of view.

21. The method as claimed in claim 17, wherein

the 3D image recording device has a second camera which is coupled to the first camera in such a way and is arranged at a distance from the first camera in such a way that the at least partly overlapping fields of view of the cameras form the field of view of the 3D image recording device in their overlap region;

and

the three-dimensional image is generated in the field of view of the 3D image recording device from the knowledge of the relative perspectives of the cameras with respect to one another by means of image processing.

22. A system for highly precisely positioning at least one object in an end position in space by means of an industrial robot, comprising:

a first industrial robot, which is adjustable into predeterminable positions;

an optical 3D image recording device, which is calibrated in a three-dimensional spatial coordinate system and is positioned in a known position with a known alignment, is designed for electronically recording three-dimensional images within a specific field of view, wherein the three-dimensional images are in each case composed of a multiplicity of pixels which are each assigned an item of depth information, has a drive unit for the alignment, which brings about adjustment of the field of view—

of the 3D image recording device, and has an angle measuring unit calibrated in the spatial coordinate system and serving for highly precisely detecting the angular alignment of the 3D image recording device, such that it is possible to determine the field of view in the spatial coordinate system; and

a control apparatus, which has a data processing device designed for image processing, wherein the control apparatus has a data connection to the first industrial robot and the optical 3D image recording device in such a way that;

the three-dimensional images recorded by the optical 3D image recording device are fed to the control apparatus;

the angular alignment of the 3D image recording device, said angular alignment being detected by the angle measuring unit, is fed to the control apparatus;

the drive unit is driven for aligning the 3D image recording device by means of the control apparatus; and

the first industrial robot is adjusted in positions that are predetermined by the control apparatus,

wherein the control apparatus and the data processing device thereof are designed in such a way that;

a first object, which has optically detectable first features known to the control apparatus, is gripped and held by the first industrial robot within a gripping tolerance;

a first compensation variable, correcting the gripping tolerance, for the first industrial robot is determined by the control apparatus such that the first object is adjustable in a compensated fashion in the spatial coordinate system by predetermining a position of the first industrial robot, wherein the first compensation variable is determined by means of the control apparatus by the steps of;

aligning the field of view of the 3D image recording device by means of the drive unit with at least one portion of the first features of the first object held in a first compensation position of the first industrial robot;

recording at least one first three-dimensional image;

determining the position of the first object in the spatial coordinate system in the first compensation position of the first industrial robot from the position of the 3D image recording device, the angular alignment of the 3D image recording device, said angular alignment being detected by the angle measuring unit, the first three-dimensional image, and the knowledge of the first features on the first object; and

determining the first compensation variable by utilizing the first compensation position of the first industrial robot, and at least the determined position of the first object in the first compensation position of the first industrial robot, and

the first object is highly precisely adjusted by the control apparatus to a first end position by the following steps, which are repeated until the first end position is reached in a predetermined tolerance;

recording at least one further first three-dimensional image;

determining the present position of the first object in the spatial coordinate system from;

the position of the 3D image recording device;

the angular alignment of the 3D image recording device, said angular alignment being detected by the angle measuring unit;

the further first three-dimensional image; and

the knowledge of the first features on the first object;

calculating the position difference between the present position of the first object and the first end position;

calculating a new desired position of the first industrial robot taking account of the first compensation variable from the present position of the first industrial robot, and a variable which is linked to the position difference and is formed, in particular, by the position difference multiplied by a factor of less than or equal to 1; and

adjusting the first industrial robot to the new desired position.

23. The system as claimed in claim 22, wherein the optical 3D image recording device is embodied in such a way that:

at least one first camera can be aligned by means of the drive unit about in each case a horizontal tilting axis with respect to the spatial coordinate system and a vertical axis; and

the angle measuring unit in each case detects the horizontal angular alignment about the vertical axis and the vertical angular alignment about the tilting axis in the spatial coordinate system.

24. The system as claimed in claim 22, wherein the optical 3D image recording device is embodied in such a way that:

at least one first camera can be aligned by means of the drive unit about in each case a horizontal tilting axis with respect to the spatial coordinate system and a vertical axis;

the angle measuring unit in each case detects the horizontal angular alignment about the vertical axis and the vertical angular alignment about the tilting axis in the spatial coordinate system; and

the horizontal tilting axis and the vertical axis substantially intersect.

25. The system as claimed in claim 24, wherein the first camera is embodied as an RIM camera for directly recording the three-dimensional image composed of a multiplicity of pixels each having assigned depth information.

26. The system as claimed in claim 24, wherein:

the 3D image recording device has an electro-optical distance measuring device; and

the control apparatus and the data processing device thereof and/or the 3D image recording device are embodied in such a way that the three-dimensional image is generated by detecting the items of depth information by means of at least one optical depth measurement on sections of the object which correspond to the pixels by means of the electro-optical distance measuring device, and assigning the items of depth information to the multiplicity of pixels.

27. The system as claimed in claim 26, wherein the electro-optical distance measuring device:

is embodied as a point scanner and the items of depth information are detected by serial scanning by means of the point scanner;

wherein at least one point-like measurement beam optically scans point by point an area of the object which corresponds to the field of view;

or

is embodied as a line scanner and the items of depth information are detected by parallel scanning by means of the line scanner, wherein at least one line-like measurement beam optically scans line by line an area of the object which corresponds to the field of view;

or

is embodied as an area depth measuring device and the items of depth information are detected by fully parallel scanning by means of the area depth measuring device, wherein at least one areal measurement beam, formed in particular by a bundle composed of a multiplicity of measurement beams, optically scans area by area an area of the object which corresponds to the field of view.