Natural media painting using a realistic brush and tablet stylus gestures

• US 9,483,138 B2
• Filed: 07/08/2010
• Issued: 11/01/2016
• Est. Priority Date: 07/10/2009
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Harris, Jerry G., DiVerdi, Stephen J., Hadap, Sunil
• Primary Examiner(s): To, Jennifer
• Assistant Examiner(s): Fortino, Ashley
• Application Number:
  US12/832,960
• Publication Number:
  US 20130125068A1
• Time in Patent Office: 2,308 Days
• Field of Search: 715/702, 715/863, 345/581
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

Systems and methods for providing a natural media painting application may receive user inputs through tablet stylus gestures. A user interface may detect stylus gestures that mimic real-world actions of artists based on information collected during user manipulation of the stylus, and may map the gestures to various digital painting and image editing tasks that may be invoked and/or controlled using the gesture-based inputs. The collected information may include spatial and/or directional information, acceleration data, an initial and/or ending position of the stylus, an initial and/or ending orientation of the stylus, and/or pressure data. The stylus gestures may include translations, rotations, twisting motions, mashing gestures, or jerking motions. The application may perform appropriate painting and image editing actions in response to detecting and recognizing the stylus gestures, and the actions taken may be dependent on the work mode and/or context of the graphics application in which stylus gesture was performed.

Claims

1. A method, comprising:

using a computer to perform;

collecting information about a user manipulation of a stylus in relation to a tablet device associated with the computer;

recognizing, from the collected information, a stylus gesture performed by the user via manipulation of the stylus, such that;

the stylus gesture is one of a plurality of stylus gestures that are recognized by the computer to perform at least one of a plurality of actions in a graphics application that comprises a natural media painting application,

at least some of the stylus gestures are mapped to user manipulation of the stylus at a distance from the tablet device,

at least some of the stylus gestures involve contact of the stylus with the tablet device, and

the stylus gestures include a brush switching gesture in which a proximity of the stylus to the tablet changes from a first position relative to the tablet to a second position that is further away from the tablet and with at least a rate of change that corresponds to the brush switching gesture, the first position being within a first pre-defined distance threshold relative the tablet and the second position being beyond a second pre-defined distance threshold, and switching between paintbrushes of a brush tool being performed responsive to recognition of the brush switching gesture;

determining which of the plurality of actions to perform based on the recognized stylus gesture; and

performing a painting function for a digital image in the graphics application including performing the determined actions.

2. The method of claim 1, wherein the tablet device is an integrated component of the computer.

3. The method of claim 1, wherein the collected information comprises information about pressure applied to the tablet device by the stylus during the manipulation of the stylus.

4. The method of claim 1, wherein the plurality of stylus gestures that are recognized by the computer includes a translation in one or more dimensions, a twisting motion, a barrel rotation, a fanning motion, a mashing down of the stylus into a tablet device, a jerking down motion, shaking the stylus away from the tablet device, and shaking the stylus toward the tablet device.

5. The method of claim 1, wherein the plurality of actions comprises one or more of:

sharpening a point of a brush tool, cleaning the brush tool, refilling the brush tool, changing a tool, drying the brush tool, splattering paint from the brush tool, adding paint to the brush tool saturating the brush tool, homogenizing the colors on the tip of the brush tool, or splitting the tip of the brush tool in the graphics application.

6. The method of claim 1, wherein the collected information comprises one or more of:

spatial or directional information collected during the manipulation of the stylus, acceleration data collected during the manipulation of the stylus, the first position of the stylus, the second position of the stylus, an initial orientation of the stylus, or an ending orientation of the stylus.

7. The method of claim 1, wherein the determined actions are determined based, at least in part, on a context in which the user manipulation of the stylus was performed.

8. The method of claim 1, wherein the plurality of stylus gestures includes shaking the stylus towards the tablet, shaking the stylus away from the tablet, and twisting the stylus about the major axis of the stylus;

and

wherein the action that is performed in response to recognition of shaking the stylus towards the tablet includes splattering paint;

wherein the action that is performed in response to recognition of shaking the stylus away from the tablet includes cleaning the brush tool; and

wherein the action that is performed in response to recognition of twisting the stylus includes homogenizing the colors of paint on the brush tool.

9. The method of claim 1, wherein the plurality of stylus gestures includes waving the stylus back and forth above the tablet, mashing the stylus into the tablet, and twisting the stylus about the major axis of the stylus;

and

wherein the action that is performed in response to recognition of waving the stylus back and forth above the tablet includes drying the brush tool;

wherein the action that is performed in response to recognition of mashing the stylus into the tablet includes splitting the bristles of the brush tool; and

wherein the action that is performed in response to recognition of twisting the stylus includes sharpening the point of the brush tool.

10. The method of claim 1, wherein the collected information includes acceleration data that is collected using an accelerometer of the stylus to augment other collected information, the acceleration data being usable to determine the rate of change for proximity changes of the stylus between the first position and the second position.

11. A system, comprising:

a stylus input module;

a display device; and

a computer device that during operation executes the stylus input module to perform operations comprising;

collecting information about a user manipulation of a stylus in relation to the computer device for both manipulation at a distance from the computer device and manipulation to provide touch input, the information including at least information regarding motion, orientation, and position of the stylus and an amount of pressure applied with the stylus;

recognizing, from the collected information, a stylus gesture performed by the user via manipulation of the stylus, wherein the stylus gesture is one of a plurality of stylus gestures that are recognized by the computer device to perform at least one of a plurality of actions in a graphics application, including changing a brush type;

determining which of the plurality of actions to perform based on the recognized stylus gesture;

wherein the changing the brush type is determined in response to recognition of a brush switching stylus gesture in which a proximity of the stylus changes from a first position relative to the display device to a second position that is further away from the display device and with at least a rate of change that corresponds to the brush switching stylus gesture, the rate of change determined using acceleration data that is collected from an accelerometer of the stylus to augment the collected information and indicates acceleration that results from jerking the stylus to change the proximity; and

performing a painting function for a digital image displayed on the display device by the graphics application including performing the determined actions.

12. The system of claim 11, further comprising:

a pressure sensitive surface;

wherein the collected information comprises information about the manipulation of the stylus in relation to the pressure sensitive surface.

13. The system of claim 11, wherein the graphics application comprises a natural media painting application, and wherein the painting function comprises a simulation of a real-world painting action using a brush tool.

14. The system of claim 11, wherein the collected information comprises one or more of:

spatial information collected during the manipulation of the stylus, directional information collected during the manipulation of the stylus, the acceleration data collected during the manipulation of the stylus, the first position of the stylus, the second position of the stylus, an initial orientation of the stylus, or an ending orientation of the stylus.

15. The system of claim 11, wherein the plurality of stylus gestures includes a barrel rotation, a fanning motion, mashing down the stylus, a jerking down motion, shaking the stylus away from a tablet device, and shaking the stylus toward the tablet.

16. A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to perform acts comprising:

collecting information about a user manipulation of a stylus indicative of six degrees of freedom movement of the stylus relative to an initial position of the stylus in three-dimensional space relative to a touch input surface and information regarding touch or pressure input applied to the touch input surface;

recognizing, from the collected information, a stylus gesture performed by the user via manipulation of the stylus, wherein the stylus gesture is one of a plurality of stylus gestures that are recognized by the one or more computers to perform at least one of a plurality of actions in a graphics application, and the plurality of stylus gestures includes a twisting motion, a waving motion above the touch input surface, a brush switching gesture in which a proximity of the stylus relative the touch input surface is changed with at least a rate of change that corresponds to the brush switching gesture and changes from the initial position relative the touch input surface that is within a first pre-defined distance threshold of the touch input surface to an ending position that is further away from the touch input surface and beyond a second pre-defined distance threshold, a quick motion toward the touch input surface, shaking the stylus directed away from the touch input surface, or shaking the stylus directed towards the touch input surface;

determining which of the plurality of actions to perform based on the recognized stylus gesture, wherein an action of switching between paintbrushes of a brush tool is performed in response to recognition of the brush switching gesture; and

performing a painting function for a digital image in the graphics application including performing the determined actions.

17. The storage medium of claim 16, wherein the collected information comprises information about the manipulation of the stylus in relation to a tablet device having the touch input surface.

18. The storage medium of claim 16, wherein the graphics application comprises a natural media painting application, and wherein the painting function comprises a simulation of a real-world painting action using the brush tool.

19. The storage medium of claim 16, wherein the collected information comprises one or more of:

spatial information collected during the manipulation of the stylus, directional information collected during the manipulation of the stylus, acceleration data collected during the manipulation of the stylus, the initial position of the stylus, the ending position of the stylus, an initial orientation of the stylus, or an ending orientation of the stylus.

20. The storage medium of claim 16, wherein the plurality of stylus gestures further includes mashing down the stylus.