Direct manipulation gestures

• US 8,429,565 B2
• Filed: 08/25/2010
• Issued: 04/23/2013
• Est. Priority Date: 08/25/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Agarawala, Anand, Dubroy, Patrick, Lesinski, Adam
• Primary Examiner(s): BAUTISTA, XIOMARA L
• Application Number:
  US12/868,398
• Publication Number:
  US 20110055773A1
• Time in Patent Office: 972 Days
• Field of Search: 715/700, 715/701, 715/702, 715/764, 715/765, 715/835, 715/863, 345/173
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

The present disclosure describes various techniques that may be implemented to execute and/or interpret manipulation gestures performed by a user on a multipoint touch input interface of a computing device. An example method includes receiving a multipoint touch gesture at a multipoint touch input interface of a computing device, wherein the multipoint touch gesture comprises a gesture that is performed with multiple touches on the multipoint touch input interface, and resolving the multipoint touch gesture into a command. The example method further includes determining at least one physical simulation effect to associate with the resolved multipoint touch gesture, and rendering a unified feedback output action in a graphical user interface of the computing device by executing the command, wherein the unified feedback output action includes at least a graphical output action incorporated with the at least one physical simulation effect in the graphical user interface.

Claims

1. A method comprising:

receiving, by a computing device, an indication of a multipoint touch gesture that is performed with multiple touches at a touch interface;

determining, by the computing device, at least one physical simulation effect associated with the multipoint touch gesture, wherein the multipoint touch gesture comprises at least one of an unscrunch gesture, a shove gesture, a pile fan-out gesture, a pile-to-grid gesture, and a crop gesture; and

outputting, by the computing device and for display at a graphical user interface, a unified feedback output action, wherein the unified feedback output action includes applying the at least one physical simulation effect to at least a graphical output action in the graphical user interface.

2. The method of claim 1, wherein the at least one physical simulation effect accounts for one or more spatial attributes of at least one of a subject item and a target item associated with at least the graphical output action.

3. The method of claim 2, wherein rendering the unified feedback output action comprises rendering the unified feedback output action to provide the at least one physical simulation effect as the subject item moves towards the target item in the graphical user interface.

4. The method of claim 2, wherein the one or more spatial attributes comprise at least one of a size attribute, a weight attribute, a mass attribute, and a location attribute.

5. The method of claim 1, wherein the at least one physical simulation effect comprises one or more of a magnetic attraction effect, a magnetic repulsion effect, a friction force effect, a dampening force effect, a centrifugal force effect, an elastic spring force effect, a vibration force effect, a speed effect, an acceleration effect, a brightened level of light effect, and a dimmed level of light effect.

6. The method of claim 5, wherein rendering the unified feedback output action comprises rendering the unified feedback output action to provide the at least one physical simulation effect as a subject item enters into a proximity of a target item in the graphical user interface.

7. The method of claim 1, further comprising associating the resolved multipoint touch gesture with multiple physical simulation effects, and wherein rendering the unified feedback output action comprises combining the multiple physical simulation effects for incorporation with at least the graphical output action.

8. The method of claim 7, wherein combining the multiple physical simulation effects comprises combining a force effect with an audio level effect.

9. The method of claim 1,

wherein receiving the multipoint touch gesture at the touch interface comprises receiving a toss gesture that includes a first touch applied at a target position associated with a target item and a second touch applied at a subject position associated with a subject item, the toss gesture further including a movement of the second touch applied at the subject position towards the target position,

wherein resolving the multipoint touch gesture comprises recognizing the toss gesture, and

wherein at least the graphical output action comprises a movement of the subject item associated with the subject position towards the target item associated with the target position in the graphical user interface.

10. The method of claim 1,

wherein receiving the multipoint touch gesture at the touch interface comprises receiving a scrunch gesture that includes multiple touches at positions that define a first bounded region encompassing a set of associated subject items displayed in the graphical user interface, the scrunch gesture further including closing of the multiple touches together to define a second bounded region,

wherein resolving the multipoint touch gesture comprises recognizing the scrunch gesture, and

wherein at least the graphical output action comprises coalescing the set of associated subject items closer together in the graphical user interface.

11. The method of claim 1,

wherein receiving the multipoint touch gesture at the touch interface comprises receiving an unscrunch gesture that includes multiple touches at positions that define a first bounded region encompassing a set of associated subject items displayed in the graphical user interface, the unscrunch gesture further including spreading of the multiple touches apart to define a second bounded region,

wherein resolving the multipoint touch gesture comprises recognizing the unscrunch gesture, and

wherein at least the graphical output action comprises spreading the set of associated subject items apart in the graphical user interface.

12. The method of claim 1,

wherein receiving the multipoint touch gesture at the touch interface comprises receiving a shove gesture that includes a side-of-the-palm contact on a continuous region of the touch interface that encompasses a set of associated subject items at least with regard to a given direction, the shove gesture further including movement of the continuous region in the given direction,

wherein resolving the multipoint touch gesture comprises recognizing the shove gesture, and

wherein at least the graphical output action comprises moving the set of associated subject items along the given direction in the graphical user interface.

13. The method of claim 1,

wherein receiving the multipoint touch gesture at the touch interface comprises receiving a pile fan-out gesture that includes multiple touches at one or more positions associated with a pile of subject pile items, the pile fan-out gesture further including movement of the multiple touches in a specified direction,

wherein resolving the multipoint touch gesture comprises recognizing the pile fan-out gesture, and

wherein at least the graphical output action comprises fanning out the subject pile items of the pile in the graphical user interface.

14. The method of claim 1,

wherein receiving the multipoint touch gesture at the touch interface comprises receiving a pile-to-grid gesture that includes multiple touches at positions encompassing an associated pile of subject items, the pile-to-grid gesture further including movement of the multiple touches continuously outward along respective disparate directions,

wherein resolving the multipoint touch gesture comprises recognizing the pile-to-grid gesture, and

wherein at least the graphical output action comprises ordering the associated subject items individually along a grid in the graphical user interface.

15. The method of claim 14, wherein the pile-to-grid gesture includes the multiple touches at four positions defining a rectangular region encompassing the associated pile of subject items, the pile-to-grid gesture further including movement of the multiple touches continuously outward along respective disparate directions to expand a size of the rectangular region.

16. The method of claim 1,

wherein receiving the multipoint touch gesture at the touch interface comprises receiving a crop gesture that includes a first touch at a position within a region associated with an object displayed in the graphical user interface, the crop gesture further including movement of a second touch to define a first crop line with respect to the object, and the crop gesture further including a release of the first touch from the position within the region associated with the object,

wherein resolving the multipoint touch gesture comprises recognizing the crop gesture, and

wherein at least the graphical output action comprises cropping the object along the first crop line in the graphical user interface.

17. The method of claim 16,

wherein the crop gesture further includes movement of the second touch to define a second crop line with respect to the object, and

wherein at least the graphical output action further comprises cropping the object along the second crop line in the graphical user interface.

18. The method of claim 17,

wherein the crop gesture further includes movement of the second touch in a specified direction from a position associated with the first or second crop line without releasing the first touch, and

wherein at least the graphical output action further comprises dragging the respective first or second crop line along the specified direction.

19. The method of claim 17,

wherein the crop gesture further includes movement of the second touch in a specified direction from a position associated with an intersection point of the first and second crop lines, and

wherein at least the graphical output action further comprises dragging both the first and second crop lines from the intersection point along the specified direction.

20. The method of claim 1, wherein the graphical user interface is part of the touch interface.

21. The method of claim 1, wherein the multipoint touch gesture comprises one or more successive single gestures.

22. The method of claim 1, wherein the multipoint touch gesture comprises multiple gestures that occur simultaneously.

23. A computer-readable storage medium comprising instructions that, when executed, cause one or more processors to:

receive an indication of a multipoint touch gesture that is performed with multiple touches at a touch interface;

determine at least one physical simulation effect to associate with the resolved multipoint touch gesture, wherein the multipoint touch gesture comprises at least one of an unscrunch gesture, a shove gesture, a pile fan-out gesture, a pile-to-grid gesture, and a crop gesture; and

output, for display at a graphical user interface, a unified feedback output action, wherein the unified feedback output action includes applying the at least one physical simulation effect to at least a graphical output action in the graphical user interface.

24. A computing device comprising:

a touch interface to receive a multipoint touch gesture that is performed with multiple touches at the touch interface;

a processor;

a physical simulation module implemented by the processor to determine at least one physical simulation effect associated with the multipoint touch gesture, wherein the multipoint touch gesture comprises at least one of an unscrunch gesture, a shove gesture, a pile fan-out gesture, a pile-to-grid gesture, and a crop gesture; and

wherein the multipoint input interface is configured to output a unified feedback output action in a graphical user interface, wherein the unified feedback action includes applying the at least one physical simulation effect to at least a graphical output action in the graphical user interface.