Natural Language Image Spatial and Tonal Localization

• US 20140081625A1
• Filed: 11/21/2012
• Published: 03/20/2014
• Est. Priority Date: 09/18/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Wilensky, Gregg D., Chang, Walter W., Dontcheva, Lubomira A., Laput, Gierad P., Agarwala, Aseem O.
• Granted Patent: US 9,412,366 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/186 Templates
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Natural language image spatial and tonal localization techniques are described. In one or more implementations, a natural language input is processed to determine spatial and tonal localization of one or more image editing operations specified by the natural language input. Performance is initiated of the one or more image editing operations on image data using the determined spatial and tonal localization.

Claims

1. A method comprising:

processing a natural language input to determine spatial and tonal localization of one or more image editing operations specified by the natural language input; and

initiating performance of the one or more image editing operations on image data using the determined spatial and tonal localization.

2. A method as described in claim 1, wherein the performance of the one or more image editing operations includes forming one or more localization masks.

3. A method as described in claim 2, wherein the one or more localization masks are a combination of a spatial localization mask with a tonal region localization mask.

4. A method as described in claim 3, wherein the spatial localization mask has spatial components that define a shape to convey a local operation, a spatial gradient function, or a border function.

5. A method as described in claim 3, wherein the tonal region localization mask has components configured to modulate tonal shapes.

6. A method as described in claim 2, wherein at least one said localization mask is formed for each filter used to perform a respective said image editing operation.

7. A method as described in claim 2, wherein a single said localization mask is formed for a composite filter operation involved in performance of a plurality of said image editing operations.

8. A method as described in claim 1, wherein the spatial localization of the natural language input is specified using at least two directions.

9. A method as described in claim 8, wherein the at least two directions include top, bottom, left, or right.

10. A method as described in claim 1, wherein at least one of the spatial localization or the tonal localization of the natural language input is specified at least in part using a gesture.

11. A method as described in claim 10, wherein the gesture is formed from a series of touch inputs that define at least part of a boundary of the portion of an image displayed using the image data.

12. A method as described in claim 10, wherein the gesture identifies a base of a portion of an image displayed using the image data that is to be subject of further processing by an object identification module to determine a boundary of the portion.

13. A method as described in claim 12, wherein the object identification module employs:

one or more facial recognition algorithms to determine the boundary of the portion;

or

one or more algorithms to identify landmarks to determine the boundary of the portion.

14. A method as described in claim 1, wherein the natural language input does not name an object included in an image displayed using the image data.

15. A method comprising:

generating one or more localization masks based on a phrase in a natural language input, each of the one or more localization masks being a combination of a spatial localization mask and a tonal region localization mask, respectively;

identifying one or more image editing operations that are included in the phrase; and

initiating performance of the one or more image editing operations on image data using the generated one or more localization masks.

16. A method as described in claim 15, wherein the spatial localization mask has spatial components that define a shape to convey a local operation, a spatial gradient function, or a border function.

17. A method as described in claim 15, wherein the tonal region localization mask has components configured to modulate tonal shapes.

18. A method as described in claim 15, wherein at least one said localization mask is a function of the image data and parameterized by a set of spatial localization parameters as well as tonal localization parameters.

19. One or more computer-readable storage media comprising instructions stored thereon that, responsive to execution on a computing device, causes the computing device to perform operations comprising:

determining a strength as well as spatial and tonal localization of one or more image editing operations specified by a natural language input, the determining of the spatial and tonal localization based at least in part of identification of a direction and a modifier included in the natural language input; and

initiating performance of the one or more image editing operations on image data using the determined spatial and tonal localization and strength.

20. One or more computer-readable storage media as described in claim 19, wherein the modifier references another direction that is substantially perpendicular to the identified direction.