Visually debugging robotic processes

• US 9,910,761 B1
• Filed: 06/28/2015
• Issued: 03/06/2018
• Est. Priority Date: 06/28/2015
• Status: Active Grant

• Current Assignee: X Development LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Jules, Anthony Sean, Jessen, Johan Ulrich Lewin
• Primary Examiner(s): Jeon, Jae
• Application Number:
  US14/752,998
• Time in Patent Office: 982 Days
• Field of Search: 717125-135
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

Methods, apparatus, systems, and computer-readable media are provided for visually debugging robotic processes. In various implementations, a graphical user interface may be rendered that includes a flowchart representing a robotic process. A plurality of different logical paths through the robotic process may be represented by a plurality of different visible paths through the flowchart. In various implementations, robot operation data indicative of one or more implementations of the robotic process may be determined. Based on the robot operation data, a first logical path through the robotic process that satisfies a criterion may be identified. In various implementations, a first visual path through the flowchart may be selected that corresponds to the identified first logical path. In various implementations, the first visible path through the flowchart may be visually distinguished from a second visible path through the flowchart.

Claims

1. A computer-implemented method, comprising:

rendering, by one or more processors, as part of a graphical user interface, a flowchart representing a robotic process, wherein a plurality of different logical paths through the robotic process are represented by a plurality of different visible paths through the flowchart;

rendering, by one or more of the processors, a robot process timeline that is visible simultaneously with the flowchart, wherein the robot process timeline represents a time interval associated with implementation of the robotic process;

implementing, by one or more of the processors, the robotic process with a real or simulated robot to generate robot operation data indicative of implementation of the robotic process, wherein the implementing is performed in response to user selection of a graphical element of the graphical user interface;

identifying, by one or more of the processors, based on the robot operation data, a first logical path through the robotic process that satisfies a criterion,

selecting, by one or more of the processors, a first visual path through the flowchart that corresponds to the identified first logical path;

determining, based on the robot operation data, a first statistic associated with the first logical path through the robotic process;

visually distinguishing, by one or more of the processors, the first visible path through the flowchart from a second visible path through the flowchart, wherein the visually distinguishing comprises rendering one or more graphical elements of the first visible path through the flowchart with a first visual characteristic selected based on the first statistic and wherein the first statistic is based on a traversal frequency on the first logical path through the robotic process;

sequentially rendering, by one or more of the processors, a graphical element at a plurality of positions along the robot process timeline in temporal synchronization with the implementation of the robotic process;

rendering, by one or more of the processors, as part of the graphical user interface, an avatar to represent a robot; and

animating, by one or more of the processors, as part of the graphical user interface, the avatar to simulate the implementation of the robotic process by the avatar, wherein animation of the avatar coincides with sequentially visually distinguishing the plurality of graphical elements of the first visible path.

2. The computer-implemented method of claim 1, wherein the criterion is satisfied by detection of an error during traversal of the first logical path.

3. The computer-implemented method of claim 1, wherein the criterion is satisfied by occurrence of an undesired result during, or as a consequence of, traversal of the first logical path.

4. The computer-implemented method of claim 1, further comprising determining, based on the robot operation data, a second statistic associated with a second logical path through the robotic process;

wherein the visually distinguishing comprises rendering one or more graphical elements of the second visible path through the flowchart with a second visual characteristic selected based on the second statistic; and

wherein the second visual characteristic is different than the first visual characteristic.

5. The computer-implemented method of claim 1, wherein

the first statistic is an error frequency associated with the first logical path.

6. The computer-implemented method of claim 1, further comprising receiving, by one or more of the processors via the graphical user interface, a user indication of the time interval, wherein determining the robot operation data comprises determining robot operation data indicative of one or more implementations of the robotic process during the time interval.

7. The computer-implemented method of claim 1, wherein

the flowchart is a first flowchart, the robotic process is a first robotic process, the robot operation data is first robot operation data, and the method further comprises:

rendering, by one or more of the processors, as part of the graphical user interface, along the robotic process timeline, a second flowchart representing a second robotic process, wherein a plurality of different visible paths through the second flowchart represent a plurality of different logical paths through the second robotic process;

implementing, by one or more of the processors, the robotic process with a real or simulated second robot to generate second robot operation data indicative of one or more implementations of the second robotic process;

identifying, by one or more of the processors, based on the second robot operation data, a first logical path through the second robotic process that satisfies a criterion;

selecting, by one or more of the processors, a first visual path through the second flowchart that corresponds to the first logical path through the second robotic process; and

visually distinguishing, by one or more of the processors, the first visible path through the second flowchart from a second visible path through the second flowchart.

8. The computer-implemented method of claim 1, wherein the visually distinguishing comprises sequentially visually distinguishing, in a manner that tracks the implementation of the robotic process along the robotic process timeline, the plurality of graphical elements of the first visible path through the flowchart.

9. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by a computing system, cause the computing system to perform the following operations:

rendering, as part of a graphical user interface, a flowchart representing a robotic process, wherein a plurality of different logical paths through the robotic process are represented by a plurality of different visible paths through the flowchart;

rendering, as part of the graphical user interface, a robot process timeline that is visible simultaneously with the flowchart, wherein the robot process timeline represents a time interval associated with implementation of the robotic process;

implementing the robotic process with a real or simulated robot to generate robot operation data indicative of implementation of the robotic process, wherein the implementing is performed in response to user selection of a graphical element of the graphical user interface;

identifying, based on the robot operation data, a first logical path through the robotic process that satisfies a criterion;

selecting a first visual path through the flowchart that corresponds to the identified first logical path;

determining, based on the robot operation data, a first statistic associated with the first logical path through the robotic process;

visually distinguishing the first visible path through the flowchart from a second visible path through the flowchart, wherein the visually distinguishing comprises rendering one or more graphical elements of the first visible path through the flowchart with a first visual characteristic selected based on the first statistic and wherein the first statistic is based on a traversal frequency on the first logical path through the robotic process;

sequentially rendering, as part of the graphical user interface, a graphical element at a plurality of positions along the robot process timeline in temporal synchronization with implementation of the robotic process;

rendering, as part of the graphical user interface that is visible simultaneously with the flowchart, an avatar that represents the real or simulated robot; and

animating, as part of the graphical user interface, the avatar to simulate the implementation of the robotic process by the avatar, wherein animation of the avatar is synchronized temporally with the implementing of the robotic process and the visually distinguishing of the first visible path through the flowchart.

10. The at least one non-transitory computer-readable medium of claim 9, wherein the rendering includes rendering, as part of the graphical user interface, one or more of a play button, a pause button, a rewind button, or a fast-forward button, wherein user selection of the play button causes the implementing, user selection of the pause button causes the implementing to be paused, user selection of the rewind button causes the implementing to be rewound, and user selection of the fast-forward button causes the implementing to be accelerated.

11. The at least one non-transitory computer-readable medium of claim 9, wherein the flow chart includes one or more nodes that are selectable to cause the implementing to initiate at a point in the robotic process that corresponds logically with the selected node.

12. A system comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:

rendering, as part of a graphical user interface, a flowchart representing a robotic process, wherein a plurality of different logical paths through the robotic process are represented by a plurality of different visible paths through the flowchart;

rendering, as part of the graphical user interface, a robot process timeline that is visible simultaneously with the flowchart, wherein the robot process timeline represents a time interval associated with implementation of the robotic process;

implementing the robotic process with a real or simulated robot to generate robot operation data indicative of implementation of the robotic process, wherein the implementing is performed in response to user selection of a graphical element of the graphical user interface;

identifying, based on the robot operation data, a first logical path through the robotic process that satisfies a criterion;

selecting a first visual path through the flowchart that corresponds to the identified first logical path;

determining, based on the robot operation data, a first statistic associated with the first logical path through the robotic process;

visually distinguishing the first visible path through the flowchart from a second visible path through the flowchart, wherein the visually distinguishing comprises rendering one or more graphical elements of the first visible path through the flowchart with a first visual characteristic selected based on the first statistic and wherein the first statistic is based on a traversal frequency on the first logical path through the robotic process;

sequentially rendering a graphical element at a plurality of positions along the robot process timeline in temporal synchronization with the implementation of the robotic process;

rendering, as part of the graphical user interface, an avatar to represent a robot; and

animating, as part of the graphical user interface, the avatar to simulate implementation of the robotic process by the avatar, wherein animation of the avatar coincides with sequentially visually distinguishing the plurality of graphical elements of the first visible path.

13. The system of claim 12, wherein the criterion is satisfied by detection of an error during traversal of the first logical path.

14. The system of claim 12, wherein the criterion is satisfied by occurrence of an undesired result during, or as a consequence of, traversal of the first logical path.