Sketch entry and interpretation of graphical user interface design

• US 10,346,476 B2
• Filed: 11/30/2018
• Issued: 07/09/2019
• Est. Priority Date: 02/05/2016
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Bequet, Henry Gabriel Victor
• Primary Examiner(s): Patel, Hiren P
• Application Number:
  US16/205,424
• Publication Number:
  US 20190114302A1
• Time in Patent Office: 221 Days
• Field of Search: None
• CPC Class Codes: G06F 16/9014 hash tables
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/043 based on fuzzy logic, fuzzy membership or fuzzy inference, e.g. adaptive neuro-fuzzy inference systems [ANFIS]
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/065 Analogue means
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

An apparatus includes a processor to employ a neural network to interpret sketch input to identify an object token that represents a command to display either details of an object or a list of objects on a specified page of a GUI. In response to identifying the object token, the processor is caused to generate GUI instructions to perform the command, and employ the neural network to further interpret the sketch input to identify text specifying a page of the GUI on which to perform the command In response to identifying the text specifying the page, the processor is caused to incorporate an indication of the page into the GUI instructions, augment a job flow definition with the GUI instructions, and store the job flow definition within a federated area in support of providing the GUI when the job flow of the job flow definition is performed.

Claims

1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:

receive first sketch input from an input device comprising a sketch of a first visual token of a set of visual tokens, wherein the first visual token is descriptive of a graphical user interface (GUI) and is interpretable to provide first GUI instructions to be executed to provide the GUI during a performance of a job flow;

employ a neural network to interpret sketch characteristics of the first sketch input to identify, from among the set of visual tokens, a first visual token represented by the first sketch input, wherein;

the neural network is trained to correlate sketch characteristics of sketch input to at least one visual token of the set of visual tokens;

the set of visual tokens comprises at least one object token that represents either a command to display details of an object of a set of objects on a specified page of the GUI, or a command to display a list of objects within the set of objects on a specified page of the GUI;

the set of objects comprises a job flow definition that defines the job flow, at least one task routine that is executed during a performance of the job flow, and at least one data object; and

the set of objects is stored within at least one federated area maintained within one or more storage devices, and to which access is authorized during the performance of the job flow; and

in response to the identification, by the neural network, of the first visual token as a first object token within the set of visual tokens, perform operations comprising;

generate first GUI instructions to perform a command correlated by the neural network to the first object token;

employ the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that specifies a page of the GUI that is to be associated with the first object token and on which the command is to be performed in providing the GUI as part of performing the job flow;

in response to identification, by the neural network, of text that specifies the page, incorporate an indication of the page as associated with the first object token into the first GUI instructions;

augment the job flow definition to include the first GUI instructions; and

store the job flow definition, following augmentation of the job flow definition to include the first GUI instructions, within a federated area of the one or more federated areas to enable access to the job flow definition with the first GUI instructions in support of the performance of the job flow and the provision of the GUI.

2. The apparatus of claim 1, wherein:

a display to visually present the first sketch input is co-located with the input device; and

the processor is caused to perform operations comprising;

visually present portions of a first sketch based on the first sketch input at a first location on the display as corresponding portions of the first sketch input are received from the input device to provide visual feedback of the first sketch input to an operator as the operator provides the first sketch input to the input device; and

following identification of at least the first visual token by the neural network, cease the visual presentation of the first sketch on the display, and visually present a graphical rendering of the first visual token in place of the first sketch at the first location to provide a visual indication to the operator of successful identification of the first visual token.

3. The apparatus of claim 1, wherein the processor is caused to perform operations including:

place the neural network into a training mode to refine the training of the neural network to correlate sketch characteristics of sketch input of a selected operator to the at least one visual token of the set of visual tokens;

receive second sketch input that originates from the selected operator;

train the neural network to correlate sketch characteristics of the second sketch input to a selected second visual token of the set of visual tokens; and

store, after completion of the training mode, neural network configuration data that defines the neural network within a federated area of the one or more federated areas to enable access to the neural network configuration data in support of employing the neural network to interpret the sketch characteristics of the first sketch input, wherein;

the neural network configuration data specifies a hyperparameter comprising at least one of a quantity of artificial neurons within the neural network or a quantity of rows of the artificial neurons into which the artificial neurons are organized within the neural network; and

the neural network configuration data also specifies a trained parameter comprising at least one of weight values of the artificial neurons, bias values of the artificial neurons or firing patterns of the artificial neurons.

4. The apparatus of claim 3, wherein:

the input device captures the entry of the first sketch input as an image of the first sketch input;

the processor is caused to provide the image of the first sketch input as an input to the neural network; and

the neural network comprises a convolution neural network that employs visual recognition to interpret the sketch characteristics of the first sketch input.

5. The apparatus of claim 3, wherein:

the input device comprises a pressure sensor to detect an amount of pressure applied during entry of the first sketch input, or a touch sensor to detect a location of a tip of a stylus or a tip of a digit of an operator during entry of the first sketch input; and

the characteristics of the first sketch input comprise at least one of;

an amount of pressure applied during entry of the first sketch input;

a pattern of change in the amount of pressure applied during entry of the first sketch input;

a velocity of movement of a point of contact with the input device during entry of the first sketch input;

an angle of a change in direction of movement of the point of contact during entry of the first sketch input;

or

a break in contact with the input device during entry of the first sketch input.

6. The apparatus of claim 1, wherein the processor is caused to, in response to the identification of the first visual token as the first object token, and in response to a lack of identification of text that specifies a page of the GUI that is to be associated with the first object token and on which the command represented by the first object token is to be performed, perform operations comprising:

identify, on a display that is co-located with the input device, a second object token that is visually presented at a second location on the display closest to a first location at which the first object token is visually presented, and that is associated with a page of the GUI on which a command represented by the second object token is to be performed; and

incorporate, into the first GUI instructions, an indication that the page that is associated with the second object token is also associated with the first object token.

7. The apparatus of claim 1, wherein the processor is caused to, in response to the identification of the first visual token as the first object token, perform operations comprising:

employ the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the sketch input that identifies the job flow definition or that identifies an instance log generated during a previous performance of the job flow definition;

in response to identification of text that identifies the job flow definition, retrieve the job flow definition from a federated area of the one or more federated areas to enable the augmentation of the job flow definition to include the first GUI instructions; and

in response to identification of text that identifies an instance log generated during a previous performance of the job flow definition, perform operations comprising;

retrieve the instance log from a federated area of the one or more federated areas;

retrieve an identifier of the job flow definition from the instance log; and

retrieve the job flow definition from a federated area of the one or more federated areas to enable the augmentation of the job flow definition to include the first GUI instructions.

8. The apparatus of claim 1, wherein the processor is caused to, in response to the identification of the first visual token as the first object token, and in response to the first object token representing a command to display details of a specific object, perform operations comprising:

employ the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that identifies the specific object; and

in response to identification of text that identifies the specific object, incorporate an indication of the specific object into the first GUI instructions prior to the augmentation of the job flow definition to include the first GUI instructions.

9. The apparatus of claim 1, wherein the processor is caused to, in response to the identification of the first visual token as the first object token, and in response to the first object token representing a command to display a list of objects, perform operations comprising:

employ the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that identifies the objects to be displayed in the list or that specifies criteria for identifying the objects to be displayed in the list; and

in response to identification of text that identifies the objects to be displayed in the list or that specifies criteria for identifying the objects to be displayed in the list, incorporate an indication of the identity of the objects to be displayed in the list or the criteria for identifying the objects to be displayed in the list into the first GUI instructions prior to augmentation of the job flow definition to include the first GUI instructions.

10. The apparatus of claim 1, wherein:

the set of visual tokens comprises at least one action token that represents either a command to jump between two specified pages of the GUI in response to receipt of a specified input, or a command to perform an action defined by an object of the set of objects in response to receipt of specified input; and

the processor is caused to, in response to the identification of the first visual token as a first action token that represents a command to jump between two specified pages of the GUI in response to receipt of at least one specified input, perform operations comprising;

generate second GUI instructions to perform the command represented by the first action token;

identify an object token that is at least partially overlain by the first action token on a display that is co-located with the input device and that visually presents the first sketch input;

incorporate, into the second GUI instructions, an indication of a page of the GUI that is associated with the overlain object token as a page of the two specified pages that is jumped from;

employ the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that specifies another page of the two specified pages that is jumped to;

incorporate, into the second GUI instructions, an indication of the page that is jumped to; and

augment the job flow definition that defines the job flow to include the second GUI instructions.

11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

receive first sketch input from an input device comprising a sketch of a first visual token of a set of visual tokens, wherein the first visual token is descriptive of a graphical user interface (GUI) and is interpretable to provide first GUI instructions to be executed to provide the GUI during a performance of a job flow;

employ a neural network to interpret sketch characteristics of the first sketch input to identify, from among the set of visual tokens, a first visual token represented by the first sketch input, wherein;

the neural network is trained to correlate sketch characteristics of sketch input to at least one visual token of the set of visual tokens;

the set of visual tokens comprises at least one object token that represents either a command to display details of an object of a set of objects on a specified page of the GUI, or a command to display a list of objects within the set of objects on a specified page of the GUI;

the set of objects comprises a job flow definition that defines the job flow, at least one task routine that is executed during a performance of the job flow, and at least one data object; and

the set of objects is stored within at least one federated area maintained within one or more storage devices, and to which access is authorized during the performance of the job flow; and

in response to the identification, by the neural network, of the first visual token as a first object token within the set of visual tokens, perform operations comprising;

generate first GUI instructions to perform a command correlated by the neural network to the first object token;

employ the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that specifies a page of the GUI that is to be associated with the first object token and on which the command is to be performed in providing the GUI as part of performing the job flow;

in response to identification, by the neural network, of text that specifies the page, incorporate an indication of the page as associated with the first object token into the first GUI instructions;

augment the job flow definition to include the first GUI instructions; and

store the job flow definition, following augmentation of the job flow definition to include the first GUI instructions, within a federated area of the one or more federated areas to enable access to the job flow definition with the first GUI instructions in support of the performance of the job flow and the provision of the GUI.

12. The computer-program product of claim 11, wherein:

a display to visually present the first sketch input is co-located with the input device; and

the processor is caused to perform operations comprising;

visually present portions of a first sketch based on the first sketch input at a first location on the display as corresponding portions of the first sketch input are received from the input device to provide visual feedback of the first sketch input to an operator as the operator provides the first sketch input to the input device; and

following identification of at least the first visual token by the neural network, cease the visual presentation of the first sketch on the display, and visually present a graphical rendering of the first visual token in place of the first sketch at the first location to provide a visual indication to the operator of successful identification of the first visual token.

13. The computer-program product of claim 11, wherein the processor is caused to perform operations including:

place the neural network into a training mode to refine the training of the neural network to correlate sketch characteristics of sketch input of a selected operator to the at least one visual token of the set of visual tokens;

receive second sketch input that originates from the selected operator;

train the neural network to correlate sketch characteristics of the second sketch input to a selected second visual token of the set of visual tokens; and

store, after completion of the training mode, neural network configuration data that defines the neural network within a federated area of the one or more federated areas to enable access to the neural network configuration data in support of employing the neural network to interpret the sketch characteristics of the first sketch input, wherein;

the neural network configuration data specifies a hyperparameter comprising at least one of a quantity of artificial neurons within the neural network or a quantity of rows of the artificial neurons into which the artificial neurons are organized within the neural network; and

the neural network configuration data also specifies a trained parameter comprising at least one of weight values of the artificial neurons, bias values of the artificial neurons or firing patterns of the artificial neurons.

14. The computer-program product of claim 13, wherein:

the input device captures the entry of the first sketch input as an image of the first sketch input;

the processor is caused to provide the image of the first sketch input as an input to the neural network; and

the neural network comprises a convolution neural network that employs visual recognition to interpret the sketch characteristics of the first sketch input.

15. The computer-program product of claim 13, wherein:

the input device comprises a pressure sensor to detect an amount of pressure applied during entry of the first sketch input, or a touch sensor to detect a location of a tip of a stylus or a tip of a digit of an operator during entry of the first sketch input; and

the characteristics of the first sketch input comprise at least one of;

an amount of pressure applied during entry of the first sketch input;

a pattern of change in the amount of pressure applied during entry of the first sketch input;

a velocity of movement of a point of contact with the input device during entry of the first sketch input;

an angle of a change in direction of movement of the point of contact during entry of the first sketch input;

or

a break in contact with the input device during entry of the first sketch input.

16. The computer-program product of claim 11, wherein the processor is caused to, in response to the identification of the first visual token as the first object token, and in response to a lack of identification of text that specifies a page of the GUI that is to be associated with the first object token and on which the command represented by the first object token is to be performed, perform operations comprising:

identify, on a display that is co-located with the input device, a second object token that is visually presented at a second location on the display closest to a first location at which the first object token is visually presented, and that is associated with a page of the GUI on which a command represented by the second object token is to be performed; and

incorporate, into the first GUI instructions, an indication that the page that is associated with the second object token is also associated with the first object token.

17. The computer-program product of claim 11, wherein the processor is caused to, in response to the identification of the first visual token as the first object token, perform operations comprising:

employ the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the sketch input that identifies the job flow definition or that identifies an instance log generated during a previous performance of the job flow definition;

in response to identification of text that identifies the job flow definition, retrieve the job flow definition from a federated area of the one or more federated areas to enable the augmentation of the job flow definition to include the first GUI instructions; and

in response to identification of text that identifies an instance log generated during a previous performance of the job flow definition, perform operations comprising;

retrieve the instance log from a federated area of the one or more federated areas;

retrieve an identifier of the job flow definition from the instance log; and

retrieve the job flow definition from a federated area of the one or more federated areas to enable the augmentation of the job flow definition to include the first GUI instructions.

18. The computer-program product of claim 11, wherein the processor is caused to, in response to the identification of the first visual token as the first object token, and in response to the first object token representing a command to display details of a specific object, perform operations comprising:

employ the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that identifies the specific object; and

in response to identification of text that identifies the specific object, incorporate an indication of the specific object into the first GUI instructions prior to the augmentation of the job flow definition to include the first GUI instructions.

19. The computer-program product of claim 11, wherein the processor is caused to, in response to the identification of the first visual token as the first object token, and in response to the first object token representing a command to display a list of objects, perform operations comprising:

employ the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that identifies the objects to be displayed in the list or that specifies criteria for identifying the objects to be displayed in the list; and

in response to identification of text that identifies the objects to be displayed in the list or that specifies criteria for identifying the objects to be displayed in the list, incorporate an indication of the identity of the objects to be displayed in the list or the criteria for identifying the objects to be displayed in the list into the first GUI instructions prior to augmentation of the job flow definition to include the first GUI instructions.

20. The computer-program product of claim 11, wherein:

the set of visual tokens comprises at least one action token that represents either a command to jump between two specified pages of the GUI in response to receipt of a specified input, or a command to perform an action defined by an object of the set of objects in response to receipt of specified input; and

the processor is caused to, in response to the identification of the first visual token as a first action token that represents a command to jump between two specified pages of the GUI in response to receipt of at least one specified input, perform operations comprising;

generate second GUI instructions to perform the command represented by the first action token;

identify an object token that is at least partially overlain by the first action token on a display that is co-located with the input device and that visually presents the first sketch input;

incorporate, into the second GUI instructions, an indication of a page of the GUI that is associated with the overlain object token as a page of the two specified pages that is jumped from;

employ the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that specifies another page of the two specified pages that is jumped to;

incorporate, into the second GUI instructions, an indication of the page that is jumped to; and

augment the job flow definition that defines the job flow to include the second GUI instructions.

21. A computer-implemented method comprising:

receiving, by a processor, first sketch input from an input device comprising a sketch of a first visual token of a set of visual tokens, wherein the first visual token is descriptive of a graphical user interface (GUI) and is interpretable to provide first GUI instructions to be executed to provide the GUI during a performance of a job flow;

employing a neural network to interpret sketch characteristics of the first sketch input to identify, from among the set of visual tokens, a first visual token represented by the first sketch input, wherein;

the neural network is trained to correlate sketch characteristics of sketch input to at least one visual token of the set of visual tokens;

the set of visual tokens comprises at least one object token that represents either a command to display details of an object of a set of objects on a specified page of the GUI, or a command to display a list of objects within the set of objects on a specified page of the GUI;

the set of objects comprises a job flow definition that defines the job flow, at least one task routine that is executed during a performance of the job flow, and at least one data object; and

the set of objects is stored within at least one federated area maintained within one or more storage devices, and to which access is authorized during the performance of the job flow; and

in response to the identification, by the neural network, of the first visual token as a first object token within the set of visual tokens, performing operations comprising;

generating first GUI instructions to perform a command correlated by the neural network to the first object token;

employing the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that specifies a page of the GUI that is to be associated with the first object token and on which the command is to be performed in providing the GUI as part of performing the job flow;

in response to identification, by the neural network, of text that specifies the page, incorporating an indication of the page as associated with the first object token into the first GUI instructions;

augmenting the job flow definition to include the first GUI instructions; and

storing the job flow definition, following augmentation of the job flow definition to include the first GUI instructions, within a federated area of the one or more federated areas to enable access to the job flow definition with the first GUI instructions in support of the performance of the job flow and the provision of the GUI.

22. The computer-implemented method of claim 21, wherein:

a display to visually present the first sketch input is co-located with the input device; and

the method comprises;

visually presenting portions of a first sketch based on the first sketch input at a first location on the display as corresponding portions of the first sketch input are received from the input device to provide visual feedback of the first sketch input to an operator as the operator provides the first sketch input to the input device; and

following identification of at least the first visual token by the neural network, ceasing the visual presentation of the first sketch on the display, and visually presenting a graphical rendering of the first visual token in place of the first sketch at the first location to provide a visual indication to the operator of successful identification of the first visual token.

23. The computer-implemented method of claim 21, comprising:

placing the neural network into a training mode to refine the training of the neural network to correlate sketch characteristics of sketch input of a selected operator to the at least one visual token of the set of visual tokens;

receiving second sketch input that originates from the selected operator;

training the neural network to correlate sketch characteristics of the second sketch input to a selected second visual token of the set of visual tokens; and

storing, after completion of the training mode, neural network configuration data that defines the neural network within a federated area of the one or more federated areas to enable access to the neural network configuration data in support of employing the neural network to interpret the sketch characteristics of the first sketch input, wherein;

the neural network configuration data specifies a hyperparameter comprising at least one of a quantity of artificial neurons within the neural network or a quantity of rows of the artificial neurons into which the artificial neurons are organized within the neural network; and

the neural network configuration data also specifies a trained parameter comprising at least one of weight values of the artificial neurons, bias values of the artificial neurons or firing patterns of the artificial neurons.

24. The computer-implemented method of claim 23, wherein:

the input device captures the entry of the first sketch input as an image of the first sketch input;

the method comprises providing the image of the first sketch input as an input to the neural network; and

the neural network comprises a convolution neural network that employs visual recognition to interpret the sketch characteristics of the first sketch input.

25. The computer-implemented method of claim 23, wherein:

the input device comprises a pressure sensor to detect an amount of pressure applied during entry of the first sketch input, or a touch sensor to detect a location of a tip of a stylus or a tip of a digit of an operator during entry of the first sketch input; and

the characteristics of the first sketch input comprise at least one of;

an amount of pressure applied during entry of the first sketch input;

a pattern of change in the amount of pressure applied during entry of the first sketch input;

a velocity of movement of a point of contact with the input device during entry of the first sketch input;

an angle of a change in direction of movement of the point of contact during entry of the first sketch input;

or

a break in contact with the input device during entry of the first sketch input.

26. The computer-implemented method of claim 21, comprising, in response to the identification of the first visual token as the first object token, and in response to a lack of identification of text that specifies a page of the GUI that is to be associated with the first object token and on which the command represented by the first object token is to be performed, performing operations comprising:

identifying, on a display that is co-located with the input device, a second object token that is visually presented at a second location on the display closest to a first location at which the first object token is visually presented, and that is associated with a page of the GUI on which a command represented by the second object token is to be performed; and

incorporating, into the first GUI instructions, an indication that the page that is associated with the second object token is also associated with the first object token.

27. The computer-implemented method of claim 21, comprising, in response to the identification of the first visual token as the first object token, performing operations comprising:

employing the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the sketch input that identifies the job flow definition or that identifies an instance log generated during a previous performance of the job flow definition;

in response to identification of text that identifies the job flow definition, retrieving the job flow definition from a federated area of the one or more federated areas to enable the augmentation of the job flow definition to include the first GUI instructions; and

in response to identification of text that identifies an instance log generated during a previous performance of the job flow definition, performing operations comprising;

retrieving the instance log from a federated area of the one or more federated areas;

retrieving an identifier of the job flow definition from the instance log; and

retrieving the job flow definition from a federated area of the one or more federated areas to enable the augmentation of the job flow definition to include the first GUI instructions.

28. The computer-implemented method of claim 21, comprising, in response to the identification of the first visual token as the first object token, and in response to the first object token representing a command to display details of a specific object, performing operations comprising:

employing the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that identifies the specific object; and

in response to identification of text that identifies the specific object, incorporating an indication of the specific object into the first GUI instructions prior to the augmentation of the job flow definition to include the first GUI instructions.

29. The computer-implemented method of claim 21, comprising, in response to the identification of the first visual token as the first object token, and in response to the first object token representing a command to display a list of objects, performing operations comprising:

employing the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that identifies the objects to be displayed in the list or that specifies criteria for identifying the objects to be displayed in the list; and

in response to identification of text that identifies the objects to be displayed in the list or that specifies criteria for identifying the objects to be displayed in the list, incorporating an indication of the identity of the objects to be displayed in the list or the criteria for identifying the objects to be displayed in the list into the first GUI instructions prior to augmentation of the job flow definition to include the first GUI instructions.

30. The computer-implemented method of claim 21, wherein:

the set of visual tokens comprises at least one action token that represents either a command to jump between two specified pages of the GUI in response to receipt of a specified input, or a command to perform an action defined by an object of the set of objects in response to receipt of specified input; and

the method comprises, in response to the identification of the first visual token as a first action token that represents a command to jump between two specified pages of the GUI in response to receipt of at least one specified input, performing operations comprising;

generating second GUI instructions to perform the command represented by the first action token;

identifying an object token that is at least partially overlain by the first action token on a display that is co-located with the input device and that visually presents the first sketch input;

incorporating, into the second GUI instructions, an indication of a page of the GUI that is associated with the overlain object token as a page of the two specified pages that is jumped from;

employing the neural network to further interpret the sketch characteristics of the first sketch input to identify text included in the first sketch input that specifies another page of the two specified pages that is jumped to;

incorporating, into the second GUI instructions, an indication of the page that is jumped to; and

augmenting the job flow definition that defines the job flow to include the second GUI instructions.