VISUAL DEBUGGING OF ROBOTIC TASKS

• US 20150239127A1
• Filed: 02/25/2014
• Published: 08/27/2015
• Est. Priority Date: 02/25/2014
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Barajas, Leandro G., Payton, David W., Ku, Li Yang, Uhlenbrock, Ryan M., Earl, Darren
• Granted Patent: US 9,387,589 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 13/026 using a predictor
• CPC Class Codes: G05B 2219/40311 Real time simulation

Abstract

A robotic system includes a robot, sensors which measure status information including a position and orientation of the robot and an object within the workspace, and a controller. The controller, which visually debugs an operation of the robot, includes a simulator module, action planning module, and graphical user interface (GUI). The simulator module receives the status information and generates visual markers, in response to marker commands, as graphical depictions of the object and robot. An action planning module selects a next action of the robot. The marker generator module generates and outputs the marker commands to the simulator module in response to the selected next action. The GUI receives and displays the visual markers, selected future action, and input commands. Via the action planning module, the position and/or orientation of the visual markers are modified in real time to change the operation of the robot.

Claims

1. A robotic system comprising:

a robot responsive to input commands;

sensors which measure a set of status information, including a position and an orientation of the robot and an object within the workspace; and

a controller having a processor and memory on which is recorded instructions for visually debugging an operation of the robot, the controller including;

a simulator module in communication with the sensors,

wherein the simulator module receives the set of status information from the sensors and generates visual markers, in response to marker commands, as graphical depictions of the object and the robot in the workspace;

an action planning module configured to select a next action of the robot;

a marker generator module in communication with the action planning module, and configured to generate and output the marker commands to the simulator module in response to the selected next action of the robot; and

a graphical user interface (GUI) having a display screen, wherein the GUI is in communication with the simulator module, and is operable to receive and display the visual markers, and to receive the input commands and modify, via the action planning module, at least one of the position and orientation of the visual markers in real time to change the operation of the robot, thereby visually debugging the operation.

2. The robotic system of claim 1, wherein the visual markers include at least one of:

a target marker indicating a desired target of an end effector of the robot, at least one trajectory marker indicating an approach or a departure trajectory of the end effector, an objective marker indicating where the object will be in the future, and a collision marker indicating where the end effector will collide with the object.

3. The robotic system of claim 2, wherein the visual markers include a pair of the trajectory markers, and wherein the pair of trajectory markers includes a first arrow indicating the approach trajectory of the end effector and a second arrow indicating the departure trajectory of the end effector.

4. The robotic system of claim 2, wherein the visual marker includes a semi-transparent shape surrounding the object that allows the object to remain visible inside of the semi-transparent shape.

5. The robotic system of claim 1, wherein the sensors include an environmental sensor operable to capture images of the object and robot, and to output the captured images to the simulator module as part of the set of status information.

6. The robotic system of claim 1, wherein the action planning module includes an action selector module and a state predictor module, wherein the action planning module selects the next action of the robot using task information from the simulator module, and the state predictor module is configured to predict future states of the robot using the task information.

7. The robotic system of claim 6, wherein the state predictor module utilizes a state tree to predict the future states.

8. The robotic system of claim 6, wherein the action planning module is configured to determine the next action as a lowest cost action using a costing model.

9. The robotic system of claim 1, wherein the simulator module includes a marker module operable to convert the marker commands into two-dimensional and three-dimensional visual markers.

10. A robotic system comprising:

a robot responsive to input commands, and having an end effector;

sensors which measure a set of status information, including a position and orientation of the robot and an object within the workspace; and

a controller having a processor and memory on which is recorded instructions for visually debugging an operation of the robot, including;

a simulator module in communication with the sensors,

wherein the simulator module receives the set of status information from the sensors and generates, in response to marker commands, a set of visual markers as graphical depictions of the object and the robot in the workspace;

an action planning module configured to select future actions of the robot as a lowest cost action using a costing model, the action planning module including an action selector module that selects a next action of the robot using task information from the simulator module;

a marker generator module in communication with the action planning module, and configured to output the marker commands to the simulator module in response to the selected next action; and

a graphical user interface (GUI) having a display screen,

wherein the GUI is in communication with the simulator module, and is operable to receive and display the visual markers and the selected future action, and also to receive the input commands and modify, via the action planning module, at least one of the position and orientation of the visual markers in real time to change the operation of the robot, thereby visually debugging the operation;

wherein the visual markers include a target marker indicating a desired target of the end effector, a trajectory marker indicating an approach trajectory arrow and a departure trajectory arrow as trajectories of the end effector, an objective marker indicating where the object will be in the future, and a collision marker indicating where the end effector will collide with the object, and wherein at least one of the target marker, the objective marker, and the collision marker includes a semi-transparent shape surrounding the object that allows the object to remain visible inside of the semi-transparent shape.

11. The robotic system of claim 10, wherein the sensors include an environmental sensor operable to capture images of the object and robot, and to output the captured images to the simulator module as part of the set of status information.

12. The robotic system of claim 10, wherein the action planning module includes a state predictor module that is configured to predict future states of the robot using the task information.

13. The robotic system of claim 12, wherein the state predictor module utilizes a state tree to predict the future states.

14. The robotic system of claim 10, wherein the simulator module includes a marker module operable to convert the marker commands into two-dimensional and three-dimensional visual markers.

15. A method for visually debugging a robot in a robotic system having a robot responsive to input commands, sensors which measure a set of status information, including a position and orientation of the robot and an object within the workspace, and a controller having a processor and memory on which is recorded instructions for visually debugging an operation of the robot, the method comprising:

receiving the set of status information from the sensors via a simulator module of the controller;

transmitting a plurality of marker commands to the simulator module via a marker generator module of the controller;

generating visual markers in response to the marker commands, via the simulator module, as graphical depictions of the object and the robot in the workspace;

displaying the visual markers and the selected future action on a display screen of a graphical user interface (GUI) of the controller;

selecting a future action of the robot via an action planning module of the controller, and

modifying, via the action planning module, at least one of the position and orientation of the visual markers in real time in response to input signals to change the operation of the robot, thereby visually debugging the operation of the robot.

16. The method of claim 15, wherein the visual markers include a target marker indicating a desired target of the end effector, a trajectory marker indicating an approach trajectory and a departure trajectory of the end effector, an objective marker indicating where the object will be in the future, and a collision marker indicating where the end effector will collide with the object.

17. The method of claim 15, wherein the action planning module includes an action selector module and a state predictor module, the method further comprising:

selecting the next action of the robot via the action planning module using task information from the simulator module; and

predicting future states of the robot via a state predictor module using the task information.

18. The method of claim 15, wherein determining the next action is performed as a lowest cost action using a costing model.

19. The method of claim 15, wherein predicting future states of the robot includes using a state tree.