Image-based trajectory robot programming planning approach

• US 10,112,303 B2
• Filed: 10/21/2014
• Issued: 10/30/2018
• Est. Priority Date: 10/25/2013
• Status: Active Grant

• Current Assignee: Aleksandar Vakanski, Farrokh Janabi-Sharifi
• Original Assignee: Aleksandar Vakanski, Farrokh Janabi-Sharifi
• Inventors: Vakanski, Aleksandar, Janabi-Sharifi, Farrokh
• Primary Examiner(s): Nguyen, Bao Long T
• Application Number:
  US15/031,779
• Publication Number:
  US 20160243704A1
• Time in Patent Office: 1,470 Days
• Field of Search: 700259, 901 3, 901 47, 31856813, 31856816, 31856818, 382153
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/36436 Arm follows movement of handheld device, camera detects, analyses motion
• CPC Class Codes: G05B 2219/36442 Automatically teaching, teach by showing
• CPC Class Codes: G05B 2219/39393 Camera detects projected image, compare with reference image, position end effector
• CPC Class Codes: G05B 2219/40116 Learn by operator observation, symbiosis, show, watch
• CPC Class Codes: G05B 2219/40607 Fixed camera to observe workspace, object, workpiece, global

Abstract

A method of programming at least one robot by demonstration comprising: performing at least one demonstration of at least one task in the Held of view of at least one fixed camera to obtain at least one observed task trajectory of at least one manipulated object, preferably at least one set of observed task trajectories; generating a generalized task trajectory from said at least one observed task trajectory, preferably from said at least one set of observed task trajectories; and executing said at least one task by said at least one robot in the field of view of said at least one fixed camera, preferably using image-based visual servoing to minimize the difference between the executed trajectory during said execution and the generalized task trajectory.

Claims

1. A method of programming at least one robot by demonstration comprising:

performing at least one set of demonstrations of at least one task in the field of view of at least one fixed camera to obtain at least one set of observed task trajectories of at least one manipulated object;

generating a generalized task trajectory from said at least one set of observed task trajectories; and

executing said at least one task by said at least one robot in the field of view of said at least one fixed camera, using image-based visual servoing to minimize the difference between an observed trajectory during said execution and the generalized task trajectory;

wherein Cartesian positions and velocities of said at least one manipulated object are calculated from image measurements from said at least one set of observed task trajectories, and said image-based visual servoing comprises following said Cartesian positions and velocities of said at least one manipulated object by minimizing the difference between said observed task trajectory during said execution and the generalized task trajectory.

2. The method of claim 1, whereas a reference task trajectory is obtained by performing a smoothing of said set of observed task trajectories.

3. The method of claim 2, whereas said smoothing is performed using a smoothing method selected from a group consisting of Kalman smoothing, hidden Markov model filtering, Gaussian mixture regression, dynamical systems smoothing and filtering.

4. The method of claim 2, whereas said generalized task trajectory is obtained by performing a second order conic optimization of the reference task trajectory within a set of constraints imposed on said set of observed task trajectories and said at least one robot.

5. The method of claim 4, whereas said constraints limit the generalized task trajectory to the field of view of said at least one camera.

6. The method of claim 4, whereas said constraints limit the generalized task trajectory to the dexterous workspace of said at least one robot.

7. The method of claim 4, whereas said constraints limit the maximum velocity of at least one end-effector of said at least one robot.

8. The method of claim 1, whereas said at least one camera further comprises at least one sensor for the perception of depth of pixels for each image acquired from said at least one camera.

9. The method of claim 8, whereas said depth of said pixels provides depth information on said set of observed task trajectories and during said image-based visual servoing.

10. The method of claim 1, whereas said at least one robot further comprises at least one tactile force sensor for monitoring tactile force during said execution of said task by said at least one robot.

11. The method of claim 1 wherein:

at least one demonstration of said at least one task is performed; and

said at least one fixed camera obtains at least one observed task trajectory by observation of said at least one demonstration.

12. A system for executing at least one task learned by observation of at least one set of demonstrations, comprising:

at least one robot with at least one movable end-effector;

at least one object to be manipulated by the at least one end-effector;

at least one fixed camera for observing said at least one set of demonstrations of the at least one task, resulting in at least one set of observed task trajectories, and observing the execution of said at least one task by said at least one robot;

at least one processing unit for generating a generalized task trajectory from said at least one set of observed task trajectories;

at least one controller for servoing said at least one robot to reproduce the task with its end-effector in the field of view of said at least one said fixed camera using image-based visual servoing to minimize the difference between the observed trajectory during the execution and the generalized task trajectory,

whereas a reference task trajectory is obtained by performing a smoothing of said set of observed task trajectories, and

whereas said generalized task trajectory is obtained by performing a second order conic optimization of the reference task trajectory within a set of constraints imposed on said observed task trajectories and said at least one robot.

13. The system of claim 12, whereas said controller and said processing unit are combined in a single computer.

14. The system of claim 12, whereas said image-based visual servoing is programmed so as to minimize the difference between the observed trajectories during said execution and the generalized task trajectory.

15. The system of claim 12, whereas said smoothing is performed using a smoothing method selected from a group consisting of Kalman smoother, hidden-Markov model smoother, Gaussian mixture regression, dynamical systems smoothing and filtering.

16. The system of claim 12, whereas said at least one robot, cameras, processing unit and controller are integrated into a single device.

17. The system of claim 12, whereas said at least one camera includes at least one sensor for perception of the depth of pixels for each image acquired from said at least one camera.

18. The system of claim 12 wherein:

said at least one task is learned by observation of at least one demonstration; and

said observation results in at least one observed task trajectory.