NATURAL LANGUAGE TEXT INSTRUCTIONS

• US 20110276944A1
• Filed: 05/07/2010
• Published: 11/10/2011
• Est. Priority Date: 05/07/2010
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Lenedev, Alexei, Pogrebisky, Michael, Hickey, Marianne, Tadeski, Inbal, Schein, Sagi, Bergman, Ruth, Pnueli, Ayelet, Barkol, Omer
• Granted Patent: US 8,756,571 B2
• Time in Patent Office: Days
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 8/38 for implementing user interfaces

Abstract

A computer displays a graphical user interface (GUI) that includes a control and receives text instructions in a natural language that describe a location of the control on the GUI. The text instructions instruct an application to perform a user interface (UI) event on the control.

Claims

1. ) A method executed by a computer, comprising:

displaying a graphical user interface (GUI) that includes a control;

receiving text instructions in a natural language that describe a location of the control on the GUI and instructs an application to perform a user interface (UI) event on the control; and

executing the UI event on the control.

2. ) The method of claim 1 further comprising:

receiving the text instructions as a set of natural language constructs; and

associating GUI semantics with the text instructions.

3. ) The method of claim 1, wherein the text instructions include identification of GUI objects that include a button, a link, a text box, and a window.

4. ) The method of claim 1, wherein the text instructions include actions of click, type, and move.

5. ) The method of claim 1, wherein the text instructions include prepositions that describe relations between objects that include left, right above, and below.

6. ) The method of claim 1 further comprising, constructing a semantic hierarchy from a GUI representation that is one of an object model representation and an image representation.

7. ) The method of claim 1 further comprising, matching an action and object description to a semantic hierarchy.

8. ) The method of claim 1 further comprising:

receiving the texts instructions at an editor;

analyzing, by the editor, the text instructions to validate semantics assigned to the text instructions.

9. ) The method of claim 1 further comprising:

simulating the text instructions on the GUI; and

recording the text instructions as an automation step; and

replaying, on a display, the automation step.

10. ) A tangible computer readable storage medium having instructions for causing a computer to execute a method, comprising:

displaying a first page of an application that includes an object;

receiving a text instruction in a natural language that both describes a location of the object on the first page and provides an instruction to execute on the object; and

executing the instruction on the object to cause the application to display a second page.

11. ) The tangible computer readable storage medium of claim 10 further comprising, executing the instruction to test a software application being developed.

12. ) The tangible computer readable storage medium of claim 10 further comprising:

receiving the text instruction in a dialogue box displayed on the first page;

executing the text instruction in a search box displayed on the first page.

13. ) The tangible computer readable storage medium of claim 10, wherein the text instruction instructs an application to click on the object and navigate to a web page of a hyperlink associated with the object.

14. ) The tangible computer readable storage medium of claim 10, wherein the text instruction provides a spatial relationship of the object with respect to another object being displayed on the first page.

15. ) The tangible computer readable storage medium of claim 10 further comprising, inferring a hierarchy of objects displayed on the first page by implementing one of image segmentation to locate components of a graphical user interface (GUI), object recognition to classify GUI components of interest, and hierarchical analysis using graph grammars to construct a hierarchy with correct semantics.

16. ) A computer system, comprising:

a display that displays a graphical user interface (GUI) with an object;

a memory storing instructions; and

a processor executing the instructions to;

receive a natural language text instruction that semantically describes a location of the object on the GUI and provides an instruction to execute on the object; and

execute the instruction on the object.

17. ) The computer system of claim 16, wherein the natural language text instruction is human spoken language.

18. ) The computer system of claim 16, wherein the natural language text instruction includes prepositions that describe a spatial relation of the object with respect to the location on a web page being displayed.

19. ) The computer system of claim 16, wherein the processor further executes the instructions to consult a thesaurus to determine a meaning of the natural language text instruction.

20. ) The computer system of claim 16, wherein the natural language text instruction is independent of any particular natural language and enables users to provide instructions in various different languages.