Systems and methods for generating functional application designs

• US 9,946,514 B1
• Filed: 07/27/2017
• Issued: 04/17/2018
• Est. Priority Date: 07/27/2017
• Status: Active Grant

• Current Assignee: HUMA.AI Co.
• Original Assignee: HUMA.AI Co.
• Inventors: Kostello, Gregory Paul
• Primary Examiner(s): Deng, Anna
• Application Number:
  US15/661,605
• Time in Patent Office: 264 Days
• Field of Search: 717105
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 8/20 Software design
• CPC Class Codes: G06F 8/30 Creation or generation of source code
• CPC Class Codes: G06F 8/35 model driven
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

A method and apparatus for generating functional application designs is described. The method may include receiving one or more natural language utterances corresponding to natural language design commands for editing an application being designed. The method may also include editing one or more components of the application being designed based on each of the natural language utterances. Furthermore, the method may include generating a functional instance of the application being designed.

Claims

1. A computer implemented method for generating functional application designs, the method comprising:

receiving, by a design tool, one or more natural language utterances corresponding to natural language design commands for editing an application being designed;

editing, by the design tool, one or more components of the application being designed based on each of the natural language utterances; and

generating, by the design tool, a functional instance of the application being designed, wherein generating the functional instance of the application being designed comprises;

analyzing, by a natural language processor, a natural language utterance from the one or more natural language utterances to extract one or more entities from the natural language utterance, wherein each of the extracted one or more entities is a component part of the natural language utterance;

determining, by a machine learning based natural language intent processor, an intent of the natural language utterance with respect to the application being designed;

determining, by the machine learning based natural language intent processor, an action that implements the determined intent in the application being designed, wherein the one or more extracted entities are variables that define the determined intent in the design command;

executing, by the design tool, the design command that creates or modifies a component in the application being designed based on the determined action; and

saving, by the design tool, the application being designed as a workspace that includes the created or modified component.

2. The method of claim 1, wherein each of the natural language utterance, the extracted one or more entities, the determined intent, the design command, the component of the application being designed, and the workspace are each associated with respective unique identifiers.

3. The method of claim 2, further comprising:

distributing the functional instance of the application being designed to at least one test system;

receiving session data indicative one or more interactions between a user of a test system and the functional instance of the application; and

storing the session data in a workspace data store.

4. The method of claim 3, further comprising:

determining a goal associated with the application being designed, wherein the goal comprises a purpose of the design, and wherein the session data includes data indicative of whether or not the goal was achieved during testing by the user of the test system;

aggregating session data for one or more applications being designed that share the goal;

performing a machine learning analysis using the aggregated session data and components of the one or more applications being designed as inputs to train a machine learning model for achieving the goal; and

determining, by the machine learning based natural language intent processor, the intent of the natural language utterance with respect to the application being designed using the trained machine learning model for achieving the goal.

5. The method of claim 3, further comprising:

aggregating session data associated with the application being designed; and

presenting the session data to a user generating the application being designed.

6. The method of claim 1, further comprising:

determining a context associated with the application begin designed, wherein the context comprises an environment in which the application being designed will be executed, a set of one or more components that make up the application being designed, a placement of one or more components, a structure of one or more pages of a design, or a combination thereof; and

determining the intent, based at least in part, on the determined context.

7. The method of claim 1, further comprising:

saving a state of the workspace for each executed design command, wherein the state comprises a transition function representing the design command that is applied to a prior state of the workspace, wherein any state of the workspace can be recreated by cumulatively applying transition functions from an initial workspace state to a desired workspace state.

8. The method of claim 1, further comprising:

receiving a natural language request to further modify the component created or modified in the application being designed based on the determined action;

storing data indicative of the determined intent as an initial intent associated with the natural language utterance;

storing data indicative of a resolved intent, wherein the resolved intent comprises an intent and zero or more entities associated with the natural language request to further modify the component, and wherein the resolved intent is further associated with the natural language utterance associated with the initial intent; and

performing a machine learning analysis using the initial intent and the resolved intent to refine a machine learning model for the analysis of intent and entity extraction from natural language utterances.

9. The method of claim 1, wherein the component comprises an abstraction of a component that defines actions and responses of the component without a definition of a visual representation of the component, and wherein the visual representation is resolved at runtime when the application being designed is executed for testing.

10. The method of claim 1, wherein a natural language utterance from the one or more natural language utterances is a verbal utterance of a user of the design tool, further comprising:

analyzing the verbal utterance with a speech to text processor; and

transforming the verbal utterance to text that corresponds to the verbal utterance.

11. The method of claim 1, wherein a natural language utterance from the one or more natural language utterances is a written utterance of a user of the design tool.

12. A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to perform a method for generating functional application designs, comprising:

receiving, by a design tool, one or more natural language utterances corresponding to natural language design commands for editing an application being designed;

editing one or more components of the application being designed based on each of the natural language utterances; and

generating, by the design tool, a functional instance of the application being designed, wherein generating the functional instance of the application being designed comprises;

analyzing, by a natural language processor, a natural language utterance from the one or more natural language utterances to extract one or more entities from the natural language utterance, wherein each of the extracted one or more entities is a component part of the natural language utterance;

determining, by a machine learning based natural language intent processor, an intent of the natural language utterance with respect to the application being designed;

determining an action that implements the determined intent in the application being designed, wherein the one or more extracted entities are variables that define the determined intent in the design command;

executing the design command that creates or modifies a component in the application being designed based on the determined action; and

saving the application being designed as a workspace that includes the created or modified component.

13. The non-transitory computer readable storage medium of claim 12, wherein each of the natural language utterance, the extracted one or more entities, the determined intent, the design command, the component of the application being designed, and the workspace are each associated with respective unique identifiers.

14. The non-transitory computer readable storage medium of claim 13, further comprising:

distributing the functional instance of the application being designed to at least one test system;

receiving session data indicative one or more interactions between a user of a test system and the functional instance of the application; and

storing the session data in a workspace data store.

15. The non-transitory computer readable storage medium of claim 14, further comprising:

determining a goal associated with the application being designed, wherein the goal comprises a purpose of the design, and wherein the session data includes data indicative of whether or not the goal was achieved during testing by the user of the test system;

aggregating session data for one or more applications being designed that share the goal;

performing a machine learning analysis using the aggregated session data and components of the one or more applications being designed as inputs to train a machine learning model for achieving the goal; and

determining, by the machine learning based natural language intent processor, the intent of the natural language utterance with respect to the application being designed using the trained machine learning model for achieving the goal.

16. The non-transitory computer readable storage medium of claim 14, further comprising:

aggregating session data associated with the application being designed; and

presenting the session data to a user generating the application being designed.

17. The non-transitory computer readable storage medium of claim 12, further comprising:

determining a context associated with the application begin designed, wherein the context comprises an environment in which the application being designed will be executed, a set of one or more components that make up the application being designed, a placement of one or more components, a structure of one or more pages of a design, or a combination thereof; and

determining the intent, based at least in part, on the determined context.

18. The non-transitory computer readable storage medium of claim 13, further comprising:

saving a state of the workspace for each executed design command, wherein the state comprises a transition function representing the design command that is applied to a prior state of the workspace, wherein any state of the workspace can be recreated by cumulatively applying transition functions from an initial workspace state to a desired workspace state.

19. The non-transitory computer readable storage medium of claim 12, further comprising:

receiving a natural language request to further modify the component created or modified in the application being designed based on the determined action;

storing data indicative of the determined intent as an initial intent associated with the natural language utterance;

storing data indicative of a resolved intent, wherein the resolved intent comprises an intent and zero or more entities associated with the natural language request to further modify the component, and wherein the resolved intent is further associated with the natural language utterance associated with the initial intent; and

performing a machine learning analysis using the initial intent and the resolved intent to refine a machine learning model for the analysis of intent and entity extraction from natural language utterances.

20. The non-transitory computer readable storage medium of claim 12, wherein the component comprises an abstraction of a component that defines actions and responses of the component without a definition of a visual representation of the component, and wherein the visual representation is resolved at runtime when the application being designed is executed for testing.

21. The non-transitory computer readable storage medium of claim 12, wherein a natural language utterance from the one or more natural language utterances is a verbal utterance of a user of the design tool, further comprising:

analyzing the verbal utterance with a speech to text processor; and

transforming the verbal utterance to text that corresponds to the verbal utterance.

22. The non-transitory computer readable storage medium of claim 12, wherein a natural language utterance from the one or more natural language utterances is a written utterance of a user of the design tool.

23. A system for generating functional application designs, the system comprising:

a memory; and

a processor coupled with the memory to;

receive one or more natural language utterances corresponding to natural language design commands for editing an application being designed,

edit one or more components of the application being designed based on each of the natural language utterances, and

generate a functional instance of the application being designed, wherein generate a functional instance of the application being designed comprises;

analyze a natural language utterance from the one or more natural language utterances to extract one or more entities from the natural language utterance, wherein each of the extracted one or more entities is a component part of the natural language utterance;

determine an intent of the natural language utterance with respect to the application being designed;

determine an action that implements the determined intent in the application being designed, wherein the one or more extracted entities are variables that define the determined intent in the design command;

execute the design command that creates or modifies a component in the application being designed based on the determined action; and

save the application being designed as a workspace that includes the created or modified component.

24. The system of claim 23, further comprising the processor to:

save a state of the workspace for each executed design command, wherein the state comprises a transition function representing the design command that is applied to a prior state of the workspace, wherein any state of the workspace can be recreated by cumulatively applying transition functions from an initial workspace state to a desired workspace state.

25. The system of claim 23, further comprising the processor to:

receive a natural language request to further modify the component created or modified in the application being designed based on the determined action;

store data indicative of the determined intent as an initial intent associated with the natural language utterance;

store data indicative of a resolved intent, wherein the resolved intent comprises an intent and zero or more entities associated with the natural language request to further modify the component, and wherein the resolved intent is further associated with the natural language utterance associated with the initial intent; and

perform a machine learning analysis using the initial intent and the resolved intent to refine a machine learning model for the analysis of intent and entity extraction from natural language utterances.