Facilitating robot positioning

• US 10,173,324 B2
• Filed: 11/16/2015
• Issued: 01/08/2019
• Est. Priority Date: 11/16/2015
• Status: Active Grant

• Current Assignee: ABB Schweiz AG (ABB Limited)
• Original Assignee: ABB Schweiz AG (ABB Limited)
• Inventors: Wang, Jianjun, Zhang, Biao, Martinez, Carlos, Morato, Carlos Winsor, Boca, Remus
• Primary Examiner(s): Kholdebarin, Iman K
• Application Number:
  US14/941,823
• Publication Number:
  US 20170136626A1
• Time in Patent Office: 1,149 Days
• Field of Search: 382153
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39057 Hand eye calibration, eye, camera on hand, end effector
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement

Abstract

Robot positioning is facilitated by obtaining, for each time of a first sampling schedule, a respective indication of a pose of a camera system of a robot relative to a reference coordinate frame, the respective indication of the pose of the camera system being based on a comparison of multiple three-dimensional images of a scene of an environment, the obtaining providing a plurality of indications of poses of the camera system; obtaining, for each time of a second sampling schedule, a respective indication of a pose of the robot, the obtaining providing a plurality of indications of poses of the robot; and determining, using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot, an indication of the reference coordinate frame and an indication of a reference point of the camera system relative to pose of the robot.

Claims

1. A method comprising:

obtaining, for each time of a plurality of times of a first sampling schedule, a respective indication of a pose of a camera system of a robot relative to a reference coordinate frame, the respective indication of the pose of the camera system being based on a comparison of multiple three-dimensional images of a scene of an environment, the obtaining providing a plurality of indications of poses of the camera system;

obtaining, for each time of a plurality of times of a second sampling schedule, a respective indication of a pose of the robot, the obtaining providing a plurality of indications of poses of the robot;

determining, using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot, an indication of the reference coordinate frame and an indication of a reference point of the camera system relative to pose of the robot; and

stitching together images of at least part of the scene of the environment using an image registration method or using measured robot positions and the determined indication of the reference point of the camera system.

2. The method of claim 1, wherein the using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot comprises building a plurality of equations relating the plurality of poses of the camera system across the first sampling schedule to the plurality of poses of the robot across the second sampling schedule, the relating being a function of the reference coordinate frame and the reference point of the camera system relative to the pose of robot, and wherein the determining comprises finding a solution for the plurality of equations to determine the indication of the reference coordinate frame and the indication of the reference point of the camera system.

3. The method of claim 2, wherein the building the plurality of equations comprises, for each indication of a pose of the camera system, of the plurality of indications of poses of the camera system, identifying a respective corresponding pose of the robot, the respective corresponding pose of the robot based on one or more indications of poses of the robot of the plurality of indications of poses of the robot.

4. The method of claim 3, wherein the first sampling schedule and second sampling schedule differ by a time-offset, and an identified respective corresponding pose of the robot for an indication of the pose of the camera system comprises an interpolated pose of the robot interpolated based on the one or more indications of poses of the robot being a pose of the robot at a time that differs from a time of obtaining one of the plurality of indications of poses of the robot by an amount of time equal to time-offset.

5. The method of claim 4, wherein the time-offset is initially unknown, and wherein the finding the solution for the plurality of equations comprises either finding a solution for the time-offset with the finding the solution for the plurality of equations to determine the indication of the reference coordinate frame and the indication of the reference point of the camera system, or first finding a solution for the time-offset and using that solution for the time-offset in finding the solution for the plurality of equations.

6. The method of claim 1, wherein pose of the robot is represented by pose of a mounting flange of the robot.

7. The method of claim 1, wherein pose of the robot is represented by joint measurements of the robot.

8. The method of claim 1, wherein the obtained plurality of indications of poses of the camera system and obtained plurality of indications of poses of the robot are obtained based on the robot obtaining images of a scene of an environment.

9. The method of claim 1, wherein the comparison is performed absent knowledge of distance between features of the scene.

10. The method of claim 1, wherein the scene comprises features of unknown characteristics.

11. The method of claim 1, wherein the determined indication of the reference point of the camera system serves as an automated calibration for the reference point of the camera system.

12. The method of claim 1, further comprising:

identifying a position of a feature in a scene space of a scene imaged by the robot;

using the determined indication of the reference coordinate frame and a relation between the determined indication of the reference coordinate frame and the identified position of the feature in the scene space to identify a position of the feature in robot space; and

providing the identified position of the feature in robot space to inform positioning of the robot.

13. The method of claim 1, wherein the obtaining the plurality of indications of poses of the camera system and the obtaining the plurality of indications of poses of the robot is associated with robot movement and imaging of a first scan region of an environment in which the robot operates, and the determining determines the indication of the reference coordinate frame and the indication of the reference point of the camera system for the first scan region, and wherein the method further comprises:

repeating the obtaining the plurality of indications of poses of the camera system and the obtaining the plurality of indications of poses of the robot in association with robot movement and imaging of at least one other scan region of the environment; and

repeating the determining to determine an indication of a reference coordinate frame and an indication of a reference point of the camera system for the at least one other scan region.

14. A computer system comprising:

a controller configured to execute program instructions to perform the acts of;

obtaining, for each time of a plurality of times of a first sampling schedule, a respective indication of a pose of a camera system of a robot relative to a reference coordinate frame, the respective indication of the pose of the camera system being based on a comparison of multiple three-dimensional images of a scene of an environment, the obtaining providing a plurality of indications of poses of the camera system;

obtaining, for each time of a plurality of times of a second sampling schedule, a respective indication of a pose of the robot, the obtaining providing a plurality of indications of poses of the robot;

determining, using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot, an indication of the reference coordinate frame and an indication of a reference point of the camera system relative to pose of the robot; and

stitching together images of at least part of the scene of the environment using an image registration method or using measured robot positions and the determined indication of the reference point of the camera system.

15. The computer system of claim 14, wherein the using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot comprises building a plurality of equations relating the plurality of poses of the camera system across the first sampling schedule to the plurality of poses of the robot across the second sampling schedule, the relating being a function of the reference coordinate frame and the reference point of the camera system relative to the pose of robot, and wherein the determining comprises finding a solution for the plurality of equations to determine the indication of the reference coordinate frame and the indication of the reference point of the camera system.

16. The computer system of claim 14, wherein the scene comprises features of unknown characteristics.

17. The computer system of claim 14, wherein the controller is configured to execute the program instructions to perform the acts of:

identifying a position of a feature in a scene space of a scene imaged by the robot;

using the determined indication of the reference coordinate frame and a relation between the determined indication of the reference coordinate frame and the identified position of the feature in the scene space to identify a position of the feature in robot space; and

providing the identified position of the feature in robot space to inform positioning of the robot.

18. A computer program product comprising:

a computer readable storage medium readable by a processor and storing instructions for execution by the processor for performing a method comprising;

obtaining, for each time of a plurality of times of a first sampling schedule, a respective indication of a pose of a camera system of a robot relative to a reference coordinate frame, the respective indication of the pose of the camera system being based on a comparison of multiple three-dimensional images of a scene of an environment, the obtaining providing a plurality of indications of poses of the camera system;

obtaining, for each time of a plurality of times of a second sampling schedule, a respective indication of a pose of the robot, the obtaining providing a plurality of indications of poses of the robot;

determining, using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot, an indication of the reference coordinate frame and an indication of a reference point of the camera system relative to pose of the robot; and

stitching together images of at least part of the scene of the environment using an image registration method or using measured robot positions and the determined indication of the reference point of the camera system.

19. The computer program product of claim 18, wherein the using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot comprises building a plurality of equations relating the plurality of poses of the camera system across the first sampling schedule to the plurality of poses of the robot across the second sampling schedule, the relating being a function of the reference coordinate frame and the reference point of the camera system relative to the pose of robot, and wherein the determining comprises finding a solution for the plurality of equations to determine the indication of the reference coordinate frame and the indication of the reference point of the camera system.

20. The computer program product of claim 18, wherein the scene comprises features of unknown characteristics.

21. A method, comprising:

obtaining a plurality of three-dimensional images of a scene of an environment;

comparing the plurality of three-dimensional images of the scene;

obtaining, for each time of a plurality of times of a first sampling schedule, a respective indication of a pose of a camera system of a robot relative to a reference coordinate frame, the respective indication of the pose of the camera system being based on the comparing of the plurality of three-dimensional images, the obtaining providing a plurality of indications of poses of the camera system;

obtaining, for each time of a plurality of times of a second sampling schedule, a respective indication of a pose of the robot, the obtaining providing a plurality of indications of poses of the robot;

determining, using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot, an indication of the reference coordinate frame and an indication of a reference point of the camera system relative to pose of the robot;

using an image registration method and also using measured robot positions and the determined indication of the reference point of the camera system to stitch together images of at least part of the scene of the environment;

identifying a position of a feature in a scene space of the scene;

determining a relation between the determined indication of the reference coordinate frame and the identified position of the feature in the scene space;

identifying the position of the feature in a robot space based on the identified position of a feature in a scene space of the scene, the determined indication of the reference coordinate frame, and the determined relation between the determined indication of the reference coordinate frame and the identified position of the feature in the scene space;

informing the position of the robot based on the identified position of the feature in the robot space; and

operating the robot based on the informed position of the robot.