Virtual Tools for Use with Touch-Sensitive Surfaces

• US 20140310631A1
• Filed: 04/15/2013
• Published: 10/16/2014
• Est. Priority Date: 04/15/2013
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Harrison, Christopher, Schwarz, Julia, Hudson, Scott E., Xiao, Robert Bo
• Granted Patent: US 10,082,935 B2
• Time in Patent Office: Days
• US Class Current: 715/771
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

An electronic device includes a touch-sensitive surface, for example a touch pad or touch screen. The user interacts with the touch-sensitive surface, producing touch interactions. Some of these touch interactions may be detected as indicative of a grasp for manipulating a physical tool (e.g., the grasp for holding a pen). When these touch interactions are encountered, a corresponding virtual tool is instantiated. The virtual tool controls an action on the electronic device that is similar to an action that can be performed by the physical tool. For example, the virtual pen can be used to draw on the display, whereas the physical pen draws on paper. A representation of the virtual tool is also displayed on a display for the electronic device, possibly providing additional affordances, at a location that corresponds to a location of the detected touch interaction.

Claims

1. A method of interaction between a user and an electronic device having a display and a touch-sensitive surface, the method comprising:

detecting a touch interaction between the user and the touch-sensitive surface that is indicative of a grasp for manipulating a physical tool; and

in response to detecting the touch interaction, instantiating a virtual tool corresponding to the physical tool, wherein the virtual tool controls an action on the electronic device that is similar to an action that can be performed by the physical tool and instantiating the virtual tool includes displaying a representation of the virtual tool at a location on the display that corresponds to a location of the detected touch interaction.

2. The method of claim 1 further comprising:

in response to detecting the touch interaction, performing the action controlled by the virtual tool.

3. The method of claim 1 further comprising:

detecting motion of the touch interaction; and

in response to detecting the motion, adjusting the representation of the virtual tool according to the detected motion and performing the action controlled by the virtual tool according to the detected motion.

4. The method of claim 1 further comprising:

after instantiating the virtual tool, detecting an additional action made by the user; and

in response to detecting the additional user action, performing the action controlled by the virtual tool according to the additional user action.

5. The method of claim 1 wherein the representation of the virtual tool is an image of the physical tool.

6. The method of claim 1 wherein the electronic device includes a touch screen and the representation of the virtual tool is displayed on the touch screen at a location of the grasp indicated by the detected touch interaction.

7. The method of claim 1 wherein the touch interaction is indicative of a grasp for grasping the physical tool lying on the touch-sensitive surface.

8. The method of claim 1 wherein the touch interaction is characterized by a touch contact pattern, and detecting the touch interaction includes determining a shape, an outline, a size and/or an orientation of the touch contact pattern.

9. The method of claim 1 wherein the touch interaction includes at least three simultaneous touch contacts, and detecting the touch interaction includes determining the number of simultaneous touch contacts in the touch interaction.

10. The method of claim 1 wherein the touch interaction includes at least three simultaneous touch contacts, and detecting the touch interaction includes determining a relative positioning of the touch contacts in the touch interaction.

11. The method of claim 1 wherein detecting the touch interaction includes determining a change in the touch interaction over time.

12. The method of claim 1 wherein the grasp is a grasp using one hand.

13. The method of claim 1 wherein the grasp is a grasp using two hands.

14. The method of claim 1 wherein the touch interaction is characterized by a touch contact pattern including at least three simultaneous touch contacts.

15. The method of claim 1 wherein the physical tool is a writing instrument, and the action controlled by the virtual tool is drawing.

16. The method of claim 1 wherein the physical tool is a paintbrush, and the action controlled by the virtual tool is digital painting.

17. The method of claim 1 wherein the physical tool is a highlighter, and the action controlled by the virtual tool is highlighting.

18. The method of claim 1 wherein the physical tool is an eraser, and the action controlled by the virtual tool is erasing.

19. The method of claim 1 wherein the physical tool is a distance measuring instrument, and the action controlled by the virtual tool is measuring a distance.

20. The method of claim 1 wherein the physical tool is a distance measuring instrument, and the action controlled by the virtual tool is displaying a distance.

21. The method of claim 1 wherein the physical tool is a cutting instrument, and the action controlled by the virtual tool is digital cutting.

22. The method of claim 1 wherein the physical tool is a camera, and the action controlled by the virtual tool is image capture.

23. The method of claim 1 wherein the physical tool is a magnifier, and the action controlled by the virtual tool is image zoom.

24. The method of claim 1 wherein the physical tool is an imprint tool, and the action controlled by the virtual tool is digital stamping.

25. The method of claim 1 wherein the physical tool is a grasping instrument, and the action controlled by the virtual tool is digital grabbing.

26. The method of claim 1 wherein the physical tool is a fastener, and the action controlled by the virtual tool is digital pinning.

27. The method of claim 1 wherein the electronic device is a phone with a touch screen.

28. The method of claim 1 wherein the electronic device is a tablet computer with a touch screen.

29. The method of claim 1 wherein the electronic device is a computer with a touch screen.

30. A machine-readable tangible storage medium having stored thereon data representing sequences of instructions, which when executed by an electronic device having a touch-sensitive surface, cause the electronic device to perform a method comprising the steps of:

detecting a touch interaction between a user and the touch-sensitive surface that is indicative of a grasp for manipulating a physical tool; and

in response to detecting the touch interaction, instantiating a virtual tool corresponding to the physical tool, wherein the virtual tool controls an action on the electronic device that is similar to an action that can be performed by the physical tool and instantiating the virtual tool includes displaying a representation of the virtual tool at a location on the display that corresponds to a location of the detected touch interaction.

31. An electronic device comprising:

a touch-sensitive surface;

detection circuitry coupled to the touch-sensitive surface, for detecting a touch interaction with the touch-sensitive surface;

a touch analysis module coupled to the detection circuitry, for analyzing the touch interaction to determine whether the touch interaction is indicative of a grasp for manipulating a physical tool; and

a processor coupled to the touch analysis module, the processor for, in response to analyzing the touch interaction, instantiating a virtual tool corresponding to the physical tool, wherein the virtual tool controls an action on the electronic device that is similar to an action that can be performed by the physical tool and instantiating the virtual tool includes displaying a representation of the virtual tool at a location on the display that corresponds to a location of the detected touch interaction.

32. An electronic device comprising:

a touch-sensitive surface;

means for detecting a touch interaction with the touch-sensitive surface;

means for analyzing the touch interaction to determine whether the touch interaction is indicative of a grasp for manipulating a physical tool; and

means for, in response to analyzing the touch interaction, instantiating a virtual tool corresponding to the physical tool, wherein the virtual tool controls an action on the electronic device that is similar to an action that can be performed by the physical tool and instantiating the virtual tool includes displaying a representation of the virtual tool at a location on the display that corresponds to a location of the detected touch interaction.