ROBOTIC TASK DEMONSTRATION INTERFACE

• US 20150331415A1
• Filed: 05/16/2014
• Published: 11/19/2015
• Est. Priority Date: 05/16/2014
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Feniello, Ashley Nathan, Birchfield, Stan, Dang, Hao
• Application Number:
  US14/280,125
• Publication Number:
  US 20150331415A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1656 characterised by programming, planning systems for manipulators
• CPC Class Codes: B25J 9/1661 characterised by task planning, object-oriented languages
• CPC Class Codes: G05B 19/425 Teaching successive positions by numerical control, i.e. commands being entered to control the positioning servo of the tool head or end effector
• CPC Class Codes: G05B 2219/36442 Automatically teaching, teach by showing
• CPC Class Codes: G05B 2219/39451 Augmented reality for robot programming
• CPC Class Codes: G05B 2219/40099 Graphical user interface for robotics, visual robot user interface
• CPC Class Codes: G05B 2219/40391 Human to robot skill transfer
• CPC Class Codes: Y10S 901/05 Machine driven lead through

Abstract

Robotic task demonstration interface embodiments are presented that generally employ a user interface to synthesize a robotic control program based on user demonstrations of object repositioning tasks, where the user manipulates objects in a displayed workspace to indicate what tasks that it is desired for a robot to perform on objects in the actual workspace associated with the robot. For example, this can involve a user repositioning objects displayed on a touch screen of a tablet computer. The configuration of the displayed workspace can be changed and additional repositioning examples performed. A robotic control program is synthesized for instructing the robot to perform the tasks indicated in the object repositioning demonstrations. The resulting learned robotic control program can be executed virtually for validation purposes, before applying it to the robot.

Claims

1. In a computer system comprising a user interface and a display device, a computer-implemented process for employing a robotics tasks demonstration interface to synthesize a robotic control program, comprising:

using a computer to perform the following process actions;

receiving object data which represents a collection of objects characterized by their orientation, location and perceptual properties in a workspace associated with a robot;

displaying a virtual rendering of the workspace in which the collection of objects are depicted so as to reflect the orientation, location and perceptual properties characterized in said object data, and in which each object displayed is capable of being repositioned so as to be depicted in a different orientation, or location, or both;

receiving user instructions to reposition one or more of the depicted objects, said object repositioning instructions being indicative of tasks that it is desired for the robot to perform on an object or objects in the actual workspace; and

synthesizing a robotic control program for instructing the robot to perform the tasks indicated in the object repositioning instructions.

2. The process of claim 1, further comprising the process actions of:

virtually executing the synthesized robotic control program to obtain changes in orientation, or location, or both of objects being repositioned by the program; and

displaying a sequence of virtual renderings of the workspace in which objects being repositioned are each shown moving from the object'"'"'s initial orientation and location to the object'"'"'s new orientation, or location, or both so as to reflect changes obtained by virtually executing the synthesized robotic control program.

3. The process of claim 2, further comprising the actions of:

receiving a user instruction indicating the changes in orientation, or location, or both of one or more objects as obtained by virtually executing the synthesized robotic control program are correct; and

designating the synthesized robotic control program to be a validated version of the synthesized robotic control program.

4. The process of claim 2, further comprising the actions of:

receiving a user instruction indicating that changes in orientation, or location, or both of one or more objects as obtained by virtually executing the synthesized robotic control program are incorrect;

designating said changes in orientation, or location, or both of one or more objects as obtained by virtually executing the synthesized robotic control program indicated as being incorrect as a negative change example;

generating negative repositioning instructions for the negative change example, said negative repositioning instructions being indicative of tasks that it is not desired for the robot to perform on an object or objects in the actual workspace; and

synthesizing a revised robotic control program for instructing the robot to perform the tasks indicated in the previously-received object repositioning instructions and not the tasks indicated in the negative prepositioning instructions.

5. The process of claim 1, further comprising the actions of:

displaying a new virtual rendering of the workspace in which the same or a modified collection of objects characterized by their orientation, location and perceptual properties are depicted;

receiving additional user instructions to reposition one or more of the objects depicted in the new virtual rendering of the workspace, said object repositioning instructions being indicative of tasks that it is desired for the robot to perform on an object or objects in the actual workspace; and

synthesizing a revised robotic control program for instructing the robot to perform the tasks indicated in the additional object repositioning instructions and the last, previously-received object repositioning instructions.

6. The process of claim 5, further comprising the process actions of:

virtually executing the revised synthesized robotic control program to obtain changes in orientation, or location, or both of objects being repositioned by the program; and

displaying a sequence of virtual renderings of the workspace in which objects being repositioned are each shown moving from the object'"'"'s initial orientation and location to the object'"'"'s new orientation, or location, or both so as to reflect changes obtained by virtually executing the revised synthesized robotic control program.

7. The process of claim 6, further comprising the actions of:

receiving a user instruction indicating the changes in orientation, or location, or both of one or more objects as obtained by virtually executing the revised synthesized robotic control program are correct; and

designating the revised synthesized robotic control program to be a new validated version of the synthesized robotic control program.

8. The process of claim 6, further comprising the actions of:

receiving a user instruction indicating that changes in orientation, or location, or both of one or more objects as obtained by virtually executing the revised synthesized robotic control program are incorrect;

designating said changes in orientation, or location, or both of one or more objects as obtained by virtually executing the revised synthesized robotic control program indicated as being incorrect as a negative change example;

generating negative repositioning instructions for the negative change example, said negative repositioning instructions being indicative of tasks that it is not desired for the robot to perform on an object or objects in the actual workspace; and

synthesizing a further revised robotic control program for instructing the robot to perform the tasks indicated in the previously-received object repositioning instructions and not the tasks indicated in the negative prepositioning instructions.

9. The process of claim 5, wherein the process action of displaying a new virtual rendering of the workspace, comprises an action of changing the orientation, or location, or both of at least one object in said collection of objects depicted in the last, previously displayed virtual rendering of the workspace.

10. The process of claim 5, wherein the process action of displaying a new virtual rendering of the workspace, comprises the actions of:

receiving new object data which represents the modified collection of objects characterized by their orientation, location and perceptual properties as depicted in an image of an actual workspace associated with a robot; and

displaying the new virtual rendering of the workspace in which the modified collection of objects are depicted so as to reflect the orientation, location and perceptual properties characterized in said new object data, and in which each object displayed is capable of being repositioned so as to be depicted in a different orientation, or location, or both.

11. In a computer system comprising a user interface and a display device, a computer-implemented process for employing a robotics tasks demonstration interface to synthesize a robotic control program, comprising:

using a computer to perform the following process actions;

receiving object data which represents a collection of objects characterized by their orientation, location and perceptual properties in a workspace associated with a robot;

displaying a virtual rendering of the workspace in which the collection of objects are depicted so as to reflect the orientation, location and perceptual properties characterized in said object data, and in which each object displayed is capable of being repositioned so as to be depicted in a different orientation, or location, or both;

receiving user instructions to reposition one or more of the depicted objects, said object repositioning instructions being indicative of tasks that it is desired for the robot to perform on an object or objects in the actual workspace;

displaying a new virtual rendering of the workspace in which the same or a modified collection of objects characterized by their orientation, location and perceptual properties are depicted;

receiving additional user instructions to reposition one or more of the objects depicted in the new virtual rendering of the workspace, said object repositioning instructions being indicative of tasks that it is desired for the robot to perform on an object or objects in the actual workspace; and

synthesizing a robotic control program for instructing the robot to perform the tasks indicated in the additional object repositioning instructions and previously-received object repositioning instructions.

12. The process of claim 11, wherein the process action of displaying a new virtual rendering of the workspace, comprises an action of changing the orientation, or location, or both of at least one object in said collection of objects depicted in the previously displayed virtual rendering of the workspace.

13. The process of claim 11, wherein the process action of displaying a new virtual rendering of the workspace, comprises the actions of:

receiving new object data which represents the modified collection of objects characterized by their orientation, location and perceptual properties as depicted in an image of an actual workspace associated with a robot; and

displaying the new virtual rendering of the workspace in which the modified collection of objects are depicted so as to reflect the orientation, location and perceptual properties characterized in said new object data, and in which each object displayed is capable of being repositioned so as to be depicted in a different orientation, or location, or both.

14. The process of claim 11, further comprising the process actions of:

virtually executing the synthesized robotic control program to obtain changes in orientation, or location, or both of objects being repositioned by the program; and

displaying a sequence of virtual renderings of the workspace in which objects being repositioned are each shown moving from the object'"'"'s initial orientation and location to the object'"'"'s new orientation, or location, or both so as to reflect changes obtained by virtually executing the synthesized robotic control program.

15. The process of claim 14, further comprising the actions of:

receiving a user instruction indicating the changes in orientation, or location, or both of one or more objects as obtained by virtually executing the synthesized robotic control program are correct; and

designating the synthesized robotic control program to be a validated version of the synthesized robotic control program.

16. The process of claim 14, further comprising the actions of:

receiving a user instruction indicating that changes in orientation, or location, or both of one or more objects as obtained by virtually executing the synthesized robotic control program are incorrect;

designating said changes in orientation, or location, or both of one or more objects as obtained by virtually executing the synthesized robotic control program indicated as being incorrect as a negative change example;

generating negative repositioning instructions for the negative change example, said negative repositioning instructions being indicative of tasks that it is not desired for the robot to perform on an object or objects in the actual workspace; and

synthesizing a revised robotic control program for instructing the robot to perform the tasks indicated in the previously-received object repositioning instructions and not the tasks indicated in the negative prepositioning instructions.

17. A system for synthesizing a robotic control program, comprising:

a computing device comprising a touch screen display; and

a robotics tasks demonstration interface computer program having program modules executable by the computing device, the computing device being directed by the program modules of the computer program to,

receive object data which represents a collection of objects characterized by their orientation, location and perceptual properties in a workspace associated with a robot,

display on said touch screen display a virtual rendering of the workspace in which the collection of objects are depicted so as to reflect the orientation, location and perceptual properties characterized in said object data, and in which each object displayed is capable of being repositioned via a touch gesture on the touch screen display so as to be depicted in a different orientation, or location, or both,

receive user instructions via the touch screen display to reposition one or more of the depicted objects, said object repositioning instructions being indicative of tasks that it is desired for the robot to perform on an object or objects in an actual workspace, and

synthesize a robotic control program for instructing the robot to perform the tasks indicated in the object repositioning instructions.

18. The system of claim 17, wherein the program module for receiving user instructions via the touch screen display, comprises sub-modules for:

sensing when an object belonging to said collection of objects displayed on the touch screen display has been touched and deeming the object to have been selected;

modifying the appearance of the selected object so as to visually distinguish it from other depicted objects;

whenever it is sensed that the selected object has been moved to a new location on the touch screen display, deeming the move to be an object repositioning instruction to reposition the selected object at said new location; and

whenever it is sensed that the selected object has been rotated to a new orientation on the touch screen display, deeming the rotation to be an object repositioning instruction to reposition the selected object at said new orientation.

19. The system of claim 17, wherein the workspace is either an image of an actual workspace associated with a robot, or a synthesized workspace.

20. The system of claim 17, wherein one or more of the objects in said collection of objects is either an actual object found in an image of an actual workspace associated with a robot, or an virtual object obtained from a database of virtual objects.