Natural Language Image Editing

• US 20140078075A1
• Filed: 11/21/2012
• Published: 03/20/2014
• Est. Priority Date: 09/18/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Wilensky, Gregg D., Chang, Walter W., Dontcheva, Lubomira A., Agarwala, Aseem O., Laput, Gierad P.
• Granted Patent: US 9,436,382 B2
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Natural language image editing techniques are described. In one or more implementations, a natural language input is converted from audio data using a speech-to-text engine. A gesture is recognized from one or more touch inputs detected using one or more touch sensors. Performance is then initiated of an operation identified from a combination of the natural language input and the recognized gesture.

Claims

1. A method implemented by one or more computing devices, the method comprising:

receiving a natural language input converted from audio data using a speech-to-text engine;

recognizing a gesture from one or more touch inputs detected using one or more touch sensors; and

initiating performance of an operation identified from a combination of the natural language input and the recognized gesture.

2. A method as described in claim 1, wherein the gesture relates to an image displayed by a display device and the operation involves editing of the image.

3. A method as described in claim 2, wherein the one or more touch sensors are configured to implement touchscreen functionality of a display device that displays the image.

4. A method as described in claim 1, further comprising determining a portion of an image that corresponds to the gesture and the performance of the operation involves the determined portion.

5. A method as described in claim 4, wherein the one or more inputs of the gesture involve a tap and the determining is performed to define at least part of a boundary of at least a portion of an image that is associated with the tap.

6. A method as described in claim 4, wherein the gesture is formed from a series of the one or more touch inputs that define at least part of a boundary of at least a portion of an image.

7. A method as described in claim 1, wherein the natural language input specifies the operation and the gesture species a subject of the performance of the operation.

8. A method as described in claim 1, wherein the gesture specifies the operation and the natural language input species a subject of the performance of the operation.

9. A method as described in claim 1, wherein at least one of the natural language input or the gesture specifies a degree to which the operation is to be performed.

10. A method comprising:

obtaining a natural language input by a computing device;

recognizing a gesture from one or more touch inputs detected using touchscreen functionality of a display device of the computing device, the display device displaying an image; and

identifying an image editing operation to be performed on the image by the computing device from a combination of the natural language input and the recognized gesture.

11. A method as described in claim 10, wherein the natural language input is converted to text from audio data using a speech-to-text engine.

12. A method as described in claim 10, wherein the natural language input includes text that is manually entered by a user.

13. A method as described in claim 10, wherein at least one of the gesture or the natural language input identifies the image editing operation and the other of the gesture or the natural language input identifies a subject of the image editing operation.

14. A method as described in claim 13, wherein the subject is a portion of the image to which the image editing operation is performed.

15. A system comprising:

a speech-to-text engine configured to convert audio data into a natural language input comprising text;

a gesture module configured to recognize a gesture from one or more touch inputs detected using one or more touch sensors, the gesture involving an image displayed by a display device; and

a natural language processing module configured to identify an operation from a combination of the natural language input and the gesture and to initiate performance of the operation by a computing device.

16. A system as described in claim 15, wherein the operation is an image editing operation.

17. A system as described in claim 15, wherein at least one of the gesture or the natural language input identifies the image editing operation and the other of the gesture or the natural language input identifies a subject of the image editing operation.

18. A system as described in claim 15, wherein at least one of the natural language input or the gesture specifies a degree to which the operation is to be performed.

19. A system as described in claim 15, wherein the speech-to-text engine is configured to form a communication for communication via a network to another speech-to-text engine responsive to a determination that at least a portion of the audio data to be converted to text does not meet a threshold, the communication including the portion of the audio data

20. A system as described in claim 15, wherein the gesture identifies a base of the image that is to be subject of further processing by an object identification module to determine a boundary of an object that includes the base.