ASSISTIVE TECHNOLOGY FOR CODE GENERATION USING VOICE AND VIRTUAL REALITY

• US 20180275957A1
• Filed: 03/27/2017
• Published: 09/27/2018
• Est. Priority Date: 03/27/2017
• Status: Abandoned Application

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Inventors: CHAVALI, Praveen, MEHER, Sunil, PITCHAIMANI, Vallinayagam, PRADHAN, Bidyapati
• Application Number:
  US15/469,770
• Publication Number:
  US 20180275957A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/635 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 8/30 Creation or generation of source code
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A method includes processing voice audio input by a user to extract a general programming instruction and determining a context of the general programming instruction based on a detected gesture (1) made by the user with respect to a display interface and (2) associated with the voice audio input by the user. The method also includes outputting an appropriate source code instruction in a programming application being displayed on the display interface based on the general programming instructions and the context.

Claims

1. A method comprising:

processing, by one or more processors, voice audio input by a user to extract a general programming instruction;

determining, by the one or more processors, a context of the general programming instruction based on a detected gesture (1) made by the user indicating an input location in a programming application displayed in a display interface and (2) associated with the voice audio input by the user;

determining an appropriate source code instruction based on the general programming instruction and the context that matches a programming language used in the programming application and a syntax of the input location; and

outputting the appropriate source code instruction in the programming application.

2. The method of claim 1,

wherein the voice audio input comprises natural language that is processed by a natural language parser to identify a particular code template of a plurality of code templates, and

wherein the voice audio input is further processed to identify values for insertion into the particular code template to create the general programming instruction.

3. The method of claim 1, wherein the appropriate source code instruction is determined by translating the general programming instruction into a programming language associated with the programming application.

4. The method of claim 1, further comprising outputting the general programming instruction for editing by the user.

5. The method of claim 1, wherein the gesture is input using a virtual reality device.

6. The method of claim 1,

wherein the voice audio is determined to be a code comment, and

wherein the code comment is output with respect to a displayed programming instruction based on the context.

7. The method of claim 1, wherein the programming application is an instance of an integrated development environment.

8. The method of claim 1, wherein the programming application is a particular one of a plurality of programming applications being displayed on the display interface.

9. The method of claim 8,

wherein the plurality of programming applications are each associated with a different programming language, and

wherein the appropriate source code instruction is suitable for a syntax of the associated programming language for the particular one of the plurality of programming applications being displayed on the display interface.

10. The method of claim 1, wherein the appropriate source code instruction is selected from a database comprising instructions in a plurality of coding languages that correspond to the general programming instruction.

11. A computer configured to access a storage device, the computer comprising:

a processor; and

a non-transitory, computer-readable storage medium storing computer-readable instructions that when executed by the processor cause the computer to perform;

processing voice audio input by a user to extract a general programming instruction;

determining a context of the general programming instruction based on a detected gesture (1) made by the user indicating an input location in a programming application displayed in a display interface and (2) associated with the voice audio input by the user;

determining an appropriate source code instruction based on the general programming instruction and the context that matches a programming language used in the programming application and a syntax of the input location; and

outputting the appropriate source code instruction in the programming application.

12. The computer of claim 11,

wherein the voice audio input comprises natural language that is processed by a natural language parser to identify a particular code template of a plurality of code templates, and

wherein the voice audio input is further processed to identify values for insertion into the particular code template to create the general programming instruction.

13. The computer of claim 11, wherein the appropriate source code instruction is determined by translating the general programming instruction into a programming language associated with the programming application.

14. The computer of claim 11, further comprising outputting the general programming instruction for editing by the user.

15. The computer of claim 11, wherein the gesture is input using a virtual reality device.

16. The computer of claim 11,

wherein the voice audio is determined to be a code comment, and

wherein the code comment is output with respect to a displayed programming instruction based on the context.

17. The computer of claim 11, wherein the programming application is an instance of an integrated development environment.

18. The computer of claim 11, wherein the programming application is a particular one of a plurality of programming applications being displayed on the display interface.

19. The computer of claim 18,

wherein the plurality of programming applications are each associated with a different programming language, and

wherein the appropriate source code instruction is suitable for a syntax of the associated programming language for the particular one of the plurality of programming applications being displayed on the display interface.

20. A non-transitory computer-readable medium having instructions stored thereon that are executable by a computing system to perform operations comprising:

processing voice audio input by a user to extract a general programming instruction;

determining a context of the general programming instruction based on a detected gesture (1) made by the user indicating an input location in a programming application displayed in a display interface and (2) associated with the voice audio input by the user;

determining an appropriate source code instruction based on the general programming instruction and the context that matches a programming language used in the programming application and a syntax of the input location; and

outputting the appropriate source code instruction in the programming application.