Natural language text instructions

• US 8,756,571 B2
• Filed: 05/07/2010
• Issued: 06/17/2014
• Est. Priority Date: 05/07/2010
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Bergman, Ruth, Barkol, Omer, Pnueli, Ayelet, Pogrebisky, Michael, Schein, Sagi, Hickey, Marianne, Tadeski, Inbal, Lenedev, Alexei
• Primary Examiner(s): Chen, Xi D
• Application Number:
  US12/776,043
• Publication Number:
  US 20110276944A1
• Time in Patent Office: 1,502 Days
• Field of Search: None
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 8/38 for implementing user interfaces

Abstract

A computer can display a graphical user interface (GUI) that includes a control. The GUI can receive text instructions in a natural language that describe a location of the control on the GUI. The text instructions can instruct an application to perform a user interface (UI) event on the control.

Claims

1. A method executed by a computer, comprising:

displaying a graphical user interface (GUI) that includes a control;

receiving text instructions in a natural language that semantically describe at least two of a location, type, and allowed action of the control on the GUI and instruct an application being developed to perform a user interface (UI) event on the control;

executing the UI event on the control, and

testing primitives in the natural language to verify automation steps produce particular results using a natural language test.

2. The method of claim 1 further comprising:

receiving the text instructions as a set of natural language constructs; and

associating GUI semantics with the text instructions.

3. The method of claim 1, wherein the text instructions include identification of GUI objects that include a button, a link, a text box, and a window.

4. The method of claim 1, wherein the text instructions include actions of click, type, and move.

5. The method of claim 1, wherein the text instructions include prepositions that describe relations between objects that include left, right above, and below.

6. The method of claim 1 further comprising, constructing a semantic hierarchy from a GUI representation that is one of an object model representation and an image representation.

7. The method of claim 1 further comprising, matching an action and object description to a semantic hierarchy.

8. The method of claim 1 further comprising:

receiving the texts instructions at an editor;

analyzing, by the editor, the text instructions to validate semantics assigned to the text instructions.

9. The method of claim 1 further comprising:

simulating the text instructions on the GUI; and

recording the text instructions as an automation step; and

replaying, on a display, the automation step.

10. A non-transitory computer readable storage medium having instructions for causing a computer to execute a method, comprising:

displaying a first page of an application being developed that includes an object;

receiving a text instruction in a natural language that semantically describes at least two of a location, type, and allowed action of the object on the first page and provides an instruction to execute on the object;

executing the instruction on the object to cause the application to display a second page, and

testing primitives in the natural language to verify automation steps produce particular results using a natural language test.

11. The non-transitory computer readable storage medium of claim 10 further comprising, executing the instruction to store language independent tests in a semantic representation.

12. The non-transitory computer readable storage medium of claim 10 further comprising:

receiving the text instruction in a dialogue box displayed on the first page;

executing the text instruction in a search box displayed on the first page.

13. The non-transitory computer readable storage medium of claim 10, wherein the text instruction instructs an application to click on the object and navigate to a web page of a hyperlink associated with the object.

14. The non-transitory computer readable storage medium of claim 10, wherein the text instruction provides a spatial relationship of the object with respect to another object being displayed on the first page.

15. The non-transitory computer readable storage medium of claim 10 further comprising, inferring a hierarchy of objects displayed on the first page by implementing one of image segmentation to locate components of a graphical user interface (GUI), object recognition to classify GUI components of interest, and hierarchical analysis using graph grammars to construct a hierarchy with correct semantics.

16. A computer system, comprising:

a display that displays a graphical user interface (GUI) with an object;

a memory storing instructions; and

a processor executing the instructions to;

receive a natural language text instruction that semantically describes at least two of a location, type, and allowed action of the object on the GUI and provides an instruction to execute on the object;

execute the instruction on the object, and

test primitives in the natural language test instruction to verify automation steps produce particular results using a natural language test.

17. The computer system of claim 16, wherein the natural language text instruction is human spoken language.

18. The computer system of claim 16, wherein the natural language text instruction includes prepositions that describe a spatial relation of the object with respect to the location on a web page being displayed.

19. The computer system of claim 16, wherein the processor further executes the instructions to consult a thesaurus to determine a meaning of the natural language text instruction.

20. The computer system of claim 16, wherein the natural language text instruction is independent of any particular natural language and enables users to provide instructions in various different languages.