Virtual tools for use with touch-sensitive surfaces

• US 10,082,935 B2
• Filed: 04/15/2013
• Issued: 09/25/2018
• Est. Priority Date: 04/15/2013
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Harrison, Christopher, Schwarz, Julia, Xiao, Robert Bo, Hudson, Scott E.
• Primary Examiner(s): Sitta, Grant
• Application Number:
  US13/863,193
• Publication Number:
  US 20140310631A1
• Time in Patent Office: 1,989 Days
• Field of Search: 345156-173
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

An electronic device includes a touch-sensitive surface, for example a touch pad or touch screen. The user interacts with the touch-sensitive surface, producing touch interactions. Some of these touch interactions may be detected as indicative of a grasp for manipulating a physical tool (e.g., the grasp for holding a pen). When these touch interactions are encountered, a corresponding virtual tool is instantiated. The virtual tool controls an action on the electronic device that is similar to an action that can be performed by the physical tool. For example, the virtual pen can be used to draw on the display, whereas the physical pen draws on paper. A representation of the virtual tool is also displayed on a display for the electronic device, possibly providing additional affordances, at a location that corresponds to a location of the detected touch interaction.

Claims

1. A method of virtual tool interaction between a user and an electronic device having a touch screen, the method comprising:

detecting a touch interaction between the user'"'"'s hand(s) and the touch screen, wherein the touch interaction is characterized by a touch contact pattern including at least three simultaneous touch contacts on the touch screen by the user'"'"'s hand(s) while the user'"'"'s hand(s) are empty but formed into a shape defined by a grasp that is suitable for manipulating a particular physical tool;

distinguishing between a first touch interaction, which has a first touch contact pattern associated with a first set of the at least three simultaneous touch contacts, and a second touch interaction, which has a second touch contact pattern associated with a second set of the at least three simultaneous touch contacts, based at least in part on at least one difference between at least one of position, shape, size, orientation, pressure, or contacting part(s) of the user'"'"'s hand(s) of the first set of the at least three simultaneous touch contacts and the second set of the at least three simultaneous touch contacts;

classifying the touch interaction as indicative of the particular physical tool based at least in part on the touch interaction being classified as any of a plurality of different touch interactions for user'"'"'s hand(s) formed into shapes defined by grasps that are suitable for manipulating the particular physical tool, wherein the plurality of different touch interactions associated with different ways for manipulating the particular physical tool are all classified as indicative of the particular physical tool, wherein the classifying the touch interaction includes classifying the touch interaction based on the distinguishing between the first touch interaction and the second touch interaction, and wherein the first touch interaction and second touch interaction correspond to different virtual tools; and

in response to classifying the touch interaction as indicative of the particular physical tool, instantiating a virtual tool corresponding to the particular physical tool, wherein the virtual tool controls an action on the electronic device that is similar to an action that can be performed by the particular physical tool and instantiating the virtual tool includes displaying a representation of the virtual tool at a location on the touch screen such that it appears the user is grasping the virtual tool.

2. The method of claim 1, further comprising:

in response to detecting the touch interaction, performing the action controlled by the virtual tool on the electronic device.

3. The method of claim 1, further comprising:

detecting motion of the touch interaction; and

in response to detecting the motion, adjusting the representation of the virtual tool based at least in part on the detecting the motion and performing the action on the electronic device based at least in part on the detecting the motion.

4. The method of claim 1, further comprising:

after instantiating the virtual tool, detecting an additional user action made by the user; and

in response to detecting the additional user action, performing the action on the electronic device based at least in part on the additional user action.

5. The method of claim 1, wherein the representation of the virtual tool is an image of the particular physical tool.

6. The method of claim 1, wherein the detecting the touch interaction includes determining at least one of a shape, an outline, a size, or an orientation of the touch contact pattern.

7. The method of claim 1, wherein the detecting the touch interaction includes determining a number of simultaneous touch contacts on the touch screen associated with the touch interaction.

8. The method of claim 1, wherein the detecting the touch interaction includes determining a relative positioning of the at least three simultaneous touch contacts on the touch screen associated with the touch interaction.

9. The method of claim 1, wherein the detecting the touch interaction includes determining a change in the touch interaction over time.

10. The method of claim 1, wherein the shape defined by the grasp that is suitable for manipulating the particular physical tool is associated with grasping the particular physical tool using one hand.

11. The method of claim 1, wherein the shape defined by the grasp that is suitable for manipulating the particular physical tool is associated with grasping the particular physical tool using two hands.

12. The method of claim 1, wherein the particular physical tool is a writing instrument, and the action on the electronic device is drawing with the virtual tool.

13. The method of claim 1, wherein the particular physical tool is a paintbrush, and the action on the electronic device is digital painting with the virtual tool.

14. The method of claim 1, wherein the particular physical tool is a highlighter, and the action is highlighting with the virtual tool.

15. The method of claim 1, wherein the particular physical tool is an eraser, and the action on the electronic device is erasing with the virtual tool.

16. The method of claim 1, wherein the particular physical tool is a distance measuring instrument, and the action on the electronic device is at least one of measuring a distance with the virtual tool or displaying the distance with the virtual tool.

17. The method of claim 1, wherein the electronic device is a display with a touch-sensitive surface.

18. The method of claim 1, wherein the particular physical tool is a cutting instrument, and the action on the electronic device is digital cutting with the virtual tool.

19. The method of claim 1, wherein the particular physical tool is a camera, and the action on the electronic device is image capture with the virtual tool.

20. The method of claim 1, wherein the particular physical tool is a magnifier, and the action on the electronic device is image zoom with the virtual tool.

21. The method of claim 1, wherein the particular physical tool is an imprint tool, and the action on the electronic device is digital stamping with the virtual tool.

22. The method of claim 1, wherein the particular physical tool is a grasping instrument, and the action on the electronic device is digital grabbing with the virtual tool.

23. The method of claim 1, wherein the particular physical tool is a fastener, and the action on the electronic device is digital pinning with the virtual tool.

24. The method of claim 1, wherein the electronic device is a phone with the touch screen.

25. The method of claim 1, wherein the electronic device is a tablet computer with the touch screen.

26. The method of claim 1, wherein the electronic device is a computer with the touch screen.

30. The method of claim 1, wherein the touch interaction is characterized by the touch contact pattern that includes a capacitive image.

31. The method of claim 1, further comprising:

learning that the plurality of different touch interactions associated with the different ways for manipulating the particular physical tool are all classified as indicative of the particular physical tool.

32. The method of claim 1, wherein each of the grasp, grasps, and different ways are all suitable for manipulating the particular physical tool.

27. A machine-readable tangible storage medium having stored thereon data representing sequences of instructions, which when executed by an electronic device having a touch screen touch sensitive surface, cause the electronic device to perform a method comprising the steps of:

detecting a touch interaction between a user'"'"'s hand(s) and the touch screen, wherein the touch interaction is characterized by a touch contact pattern including at least three simultaneous touch contacts on the touch screen by the user'"'"'s hand(s) while the user'"'"'s hand(s) are empty but formed into a shape defined by a grasp that is suitable for manipulating a particular physical tool;

distinguishing between a first touch interaction, which has a first touch contact pattern associated with a first set of the at least three simultaneous touch contacts, and a second touch interaction, which has a second touch contact pattern associated with a second set of the at least three simultaneous touch contacts, based at least in part on at least one difference between at least one of position, shape, size, orientation, pressure, or contacting part(s) of the user'"'"'s hand(s) of the first set of the at least three simultaneous touch contacts and the second set of the at least three simultaneous touch contacts;

classifying the touch interaction as indicative of the particular physical tool based at least in part on the touch interaction being classified as any of a plurality of different touch interactions for user'"'"'s hand(s) formed into shapes defined by grasps that are suitable for manipulating the particular physical tool, wherein the plurality of different touch interactions associated with different ways for manipulating the particular physical tool are all classified as indicative of the particular physical tool, wherein the classifying the touch interaction includes classifying the touch interaction based on the distinguishing between the first touch interaction and the second touch interaction, and wherein the first touch interaction and second touch interaction correspond to different virtual tools; and

in response to classifying the touch interaction as indicative of the particular physical tool, instantiating a virtual tool corresponding to the particular physical tool, wherein the virtual tool controls an action on the electronic device that is similar to an action that can be performed by the particular physical tool and instantiating the virtual tool includes displaying a representation of the virtual tool at a location on the touch screen such that it appears the user is grasping the virtual tool.

28. An electronic device comprising:

a touch screen;

detection circuitry coupled to the touch screen, for detecting a touch interaction between a user'"'"'s hand(s) and the touch screen, wherein the touch interaction is characterized by a touch contact pattern including at least three simultaneous touch contacts on the touch screen by the user'"'"'s hand(s) while the user'"'"'s hand(s) are empty but formed into a shape defined by a grasp that is suitable for manipulating a particular physical tool;

a touch analysis module coupled to the detection circuitry, for distinguishing between a first touch interaction, which has a first touch contact pattern associated with a first set of the at least three simultaneous touch contacts, and a second touch interaction, which has a second touch contact pattern associated with a second set of the at least three simultaneous touch contacts, based at least in part on at least one difference between at least one of position, shape, size, orientation, pressure, or contacting part(s) of the user'"'"'s hand(s) of the first set of the at least three simultaneous touch contacts and the second set of the at least three simultaneous touch contacts, and for classifying the touch interaction as indicative of the particular physical tool based at least in part on the touch interaction being classified as any of a plurality of different touch interactions for user'"'"'s hand(s) formed into shapes defined by grasps that are suitable for manipulating the particular physical tool, wherein the plurality of different touch interactions associated with different ways for manipulating the particular physical tool are all classified as indicative of the particular physical tool, wherein the classifying the touch interaction includes classifying the touch interaction based on the distinguishing between the first touch interaction and the second touch interaction, and wherein the first touch interaction and second touch interaction correspond to different virtual tools; and

a processor coupled to the touch analysis module, the processor for, in response to classifying the touch interaction as indicative of the particular physical tool, instantiating a virtual tool corresponding to the particular physical tool, wherein the virtual tool controls an action on the electronic device that is similar to an action that can be performed by the particular physical tool and instantiating the virtual tool includes displaying a representation of the virtual tool at a location on the touch screen such that it appears the user is grasping the virtual tool.

29. An electronic device comprising:

a touch screen;

means for detecting a touch interaction between the user'"'"'s hand(s) and the touch screen, wherein the touch interaction is characterized by a touch contact pattern including at least three simultaneous touch contacts on the touch screen by the user'"'"'s hand(s) while the user'"'"'s hand(s) are empty but formed into a shape defined by a grasp that is suitable for manipulating a particular physical tool;

means for analyzing the touch interaction to distinguish between a first touch interaction, which has a first touch contact pattern associated with a first set of the at least three simultaneous touch contacts, and a second touch interaction, which has a second touch contact pattern associated with a second set of the at least three simultaneous touch contacts, based at least in part on at least one difference between at least one of position, shape, size, orientation, pressure, or contacting part(s) of the user'"'"'s hand(s) of the first set of the at least three simultaneous touch contacts and the second set of the at least three simultaneous touch contacts, and to classify the touch interaction as indicative of the particular physical tool based at least in part on the touch interaction being classified as any of a plurality of different touch interactions for user'"'"'s hand(s) formed into shapes defined by grasps that are suitable for manipulating the particular physical tool, wherein the plurality of different touch interactions associated with different ways for manipulating the particular physical tool are all classified as indicative of the particular physical tool, and wherein the means for analyzing includes means for analyzing the touch interaction based on distinguishing between the first touch interaction and the second touch interaction, and wherein the first touch interaction and second touch interaction correspond to different virtual tools; and

means for, in response to classifying the touch interaction as indicative of the particular physical tool, instantiating a virtual tool corresponding to the particular physical tool, wherein the virtual tool controls an action on the electronic device that is similar to an action that can be performed by the particular physical tool and instantiating the virtual tool includes displaying a representation of the virtual tool at a location on the touch screen such that it appears the user is grasping the virtual tool.