Natural Language and User Interface Controls

• US 20140082500A1
• Filed: 11/21/2012
• Published: 03/20/2014
• Est. Priority Date: 09/18/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Wilensky, Gregg D., Chang, Walter W., Dontcheva, Lubomira A., Laput, Gierad P., Agarwala, Aseem O.
• Granted Patent: US 10,656,808 B2
• Time in Patent Office: Days
• US Class Current: 715/727
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

Natural language and user interface control techniques are described. In one or more implementations, a natural language input is received that is indicative of an operation to be performed by one or more modules of a computing device. Responsive to determining that the operation is associated with a degree to which the operation is performable, a user interface control is output that is manipulable by a user to control the degree to which the operation is to be performed.

Claims

1. A method comprising:

receiving a natural language input that is indicative of an operation to be performed by one or more modules of a computing device; and

responsive to determining that the operation is associated with a degree to which the operation is performable, outputting a user interface control that is manipulable by a user to control the degree to which the operation is to be performed.

2. A method as described in claim 1, wherein the operation involves editing of an image.

3. A method as described in claim 1, wherein the user interface control includes an indication of the degree to which the performance of the operation is currently set.

4. A method as described in claim 3, wherein the user interface control includes a plurality of additional indications corresponding to a plurality of the degrees to which the operation is performable, the plurality of additional indications displayed concurrently with the indication of the degree to which the performance of the operation is currently set.

5. A method as described in claim 4, wherein the user interface control is configured as a slider control or a radial dial.

6. A method as described in claim 1, wherein the user interface control is manipulable by a user using a speech input.

7. A method as described in claim 1, further comprising ceasing the outputting of the user interface control after a threshold amount of time has passed during which interaction with the user interface control has not occurred.

8. A method as described in claim 1, further comprising:

receiving audio data captured using an audio capture device; and

converting the audio data into text that forms at least part of the natural language input.

9. A method as described in claim 1, further comprising locating the user interface control from a plurality of user interface controls that is associated with the operation.

10. A method as described in claim 1, further comprising receiving one or more inputs generated through interaction with the user interface control and adjusting the degree to which the operation is performed based on the one or more inputs.

11. A method as described in claim 1, further comprising setting the degree to which the operation is to be performed to a default level responsive to a determination that the natural language user input does not specify the degree.

12. A method comprising:

displaying a user interface control that is indicative of a degree to which an image editing operation is performed on at least a portion of an image, the displaying performed;

responsive to receipt of a natural language input that specifies the image editing operation; and

simultaneously with the portion of the image to which the image editing operation is performed; and

adjusting the degree to which the image editing operation is performed responsive to one or more inputs that are received via interaction with the user interface control.

13. A method as described in claim 12, further comprising displaying a result of the adjusting on at least the portion of the image simultaneously with a display of the user interface control that is indicative of the adjusted degree.

14. A method as described in claim 13, wherein the displaying is performed in real time.

15. A method as described in claim 12, wherein the one or more inputs are generated using text, a gesture, or a speech input.

16. A method as described in claim 12, further comprising receiving audio data captured using an audio capture device and converting the audio data into text that forms at least part of the natural language input using a speech-to-text engine.

17. One or more computer-readable storage media comprising instructions stored thereon that, responsive to execution by a computing device, causes the computing device to perform operations comprising:

determining that an operation specified by a natural language input is associated with a degree to which the operation is performable;

finding which of a plurality of user interface controls are associated with the operation, the found user interface control configured to be manipulable to a user to adjust the degree to which the operation is to be performed; and

causing the user interface control to be displayed by a display device.

18. One or more computer-readable storage media as described in claim 17, wherein the user interface control is a slider.

19. One or more computer-readable storage media as described in claim 17, wherein the user interface control defines a boundary within a user interface in which the operation is to be performed.

20. One or more computer-readable storage media as described in claim 17, wherein the user interface control is displayed separate from a portion of the user interface on which the operation is to be performed.