Multi-touch marking menus and directional chording gestures

• US 9,405,404 B2
• Filed: 03/26/2010
• Issued: 08/02/2016
• Est. Priority Date: 03/26/2010
• Status: Active Grant

• Current Assignee: Autodesk, Inc.
• Original Assignee: Autodesk, Inc.
• Inventors: Fitzmaurice, George, Grossman, Tovi, Lepinski, Gerard Julian
• Primary Examiner(s): Hicks, Charles V
• Application Number:
  US12/732,978
• Publication Number:
  US 20110234503A1
• Time in Patent Office: 2,321 Days
• Field of Search: 345/173
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0235 using chord techniques G06F3/0234 takes precedence
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A method for interacting with a multi-touch surface to perform an operation. The method includes detecting that at least one finger is in contact with the multi-touch surface, identifying a chord defined by a combination of fingers associated with the at least one finger in contact with the multi-touch surface, and performing a first function associated with the operation based on the chord. The method may also include integrating the chords into directional movement gestures and using the gestures to control system functionality.

Claims

1. A method for interacting with a multi-touch surface to perform an operation, the method comprising:

determining that five fingers are in contact with the multi-touch surface;

detecting that one or more of the five fingers have been lifted away from the multi-touch surface;

after the one or more of the five fingers have been lifted away from the multi-touch surface, determining that at least one finger of the five fingers is remaining in contact with the multi-touch surface;

identifying a first chord that corresponds to the one or more of the five fingers being lifted away from the multi-touch surface and the at least one finger remaining in contact with the multi-touch surface;

in response to identifying the first chord, performing a first operation associated with the first chord.

2. The method of claim 1, wherein the step of identifying the first chord comprises:

capturing one or more images of the at least one finger remaining in contact with the multi-touch surface;

determining a bounding box within the one or more images that encloses the at least one finger remaining first plurality of fingers that is in contact with the multi-touch surface; and

analyzing the bounding box to identify the first chord associated with the at least one finger remaining in contact with the multi-touch surface.

3. The method of claim 2, wherein the one or more images are captured from beneath the multi-touch surface.

4. The method of claim 1, wherein performing the first operation comprises displaying a first plurality of user interface elements associated with the first chord on the multi-touch surface.

5. The method of claim 4, wherein the first plurality of user interface elements comprises a radial marking menu.

6. The method of claim 5, wherein the radial marking menu is offset by a fixed distance from a location at which the at least one finger is in contact with the multi-touch surface.

7. The method of claim 4, further comprising receiving a first selection of a first user interface element included in the first plurality of user interface elements, and based on receiving the first selection, executing a first application command associated with the first user interface element.

8. The method of claim 7, wherein receiving the first selection of the first user interface element comprises detecting a chording gesture caused by the at least one finger sliding along the multi-touch surface.

9. The method of claim 7, further comprising:

receiving a second selection of a second user interface element included in the first plurality of user interface elements; and

based on receiving the second selection of the second user interface element, displaying a nested marking menu comprising a second plurality of user interface elements.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause a computing device to interact with a multi-touch surface to perform an operation, by performing the steps of:

determining that five fingers are in contact with the multi-touch surface;

detecting that one or more of the five fingers have been lifted away from the multi-touch surface;

after the one or more of the five fingers have been lifted away from the multi-touch surface, determining that at least one finger of the five fingers is remaining in contact with the multi-touch surface;

identifying a first chord that corresponds to the one or more of the five fingers being lifted away from the multi-touch surface and the at least one finger remaining in contact with the multi-touch surface;

in response to identifying the first chord, performing a first operation associated with the first chord.

11. The non-transitory computer-readable storage medium of claim 10, wherein performing the first operation comprises displaying a first plurality of user interface elements associated with the first chord on the multi-touch surface.

12. The non-transitory computer-readable storage medium of claim 11, wherein the first plurality of user interface elements comprises a radial marking menu.

13. The non-transitory computer-readable storage medium of claim 12, wherein the radial marking menu is offset by a fixed distance from a location at which the at least one finger is in contact with the multi-touch surface.

14. The non-transitory computer-readable storage medium of claim 11, further comprising the steps of receiving a first selection of a first user interface element included in the first plurality of user interface elements, and based on receiving the first selection, executing a first application command associated with the first user interface element.

15. The non-transitory computer-readable storage medium of claim 14, wherein the step of receiving the first selection of the first user interface element comprises detecting a chording gesture caused by the at least one finger sliding along the multi-touch surface.

16. The non-transitory computer-readable storage medium of claim 10, wherein the step of identifying the first chord comprises:

capturing one or more images of the at least one finger remaining first plurality of in contact with the multi-touch surface;

determining a bounding box within the one or more images that encloses the at least one finger remaining in contact with the multi-touch surface; and

analyzing the bounding box to identify the first chord associated with the at least one finger remaining in contact with the multi-touch surface.

17. The non-transitory computer-readable storage medium of claim 16, wherein the one or more images are captured from beneath the multi-touch surface.

18. The non-transitory computer-readable storage medium of claim 10, further comprising the steps of:

receiving a third selection of a third user interface element included in the first plurality of user interface elements; and

based on receiving the third selection of the third user interface element, displaying a nested marking menu comprising a third plurality of user interface elements.

19. The non-transitory computer-readable storage medium of claim 10, further comprising, after determining that the five fingers are in contact with the multi-touch surface, generating a finger coordinate map that comprises finger coordinates for each of the five fingers, wherein determining that the at least one finger is remaining in contact with the multi-touch surface is performed based on the finger coordinate map.

20. A computer system, comprising:

a processor; and

a memory storing instructions that when executed by the processor cause the computer system to interact with a multi-touch surface to perform an operation, by performing the steps of;

determining that five fingers are in contact with the multi-touch surface;

detecting that one or more of the five fingers have been lifted away from the multi-touch surface;

after the one or more of the five fingers have been lifted away from the multi-touch surface, determining that at least one finger of the five fingers is remaining in contact with the multi-touch surface;

identifying a first chord that corresponds to the one or more of the five fingers being lifted away from the multi-touch surface and the at least one finger remaining in contact with the multi-touch surface;

in response to identifying the first chord, performing a first operation associated with the first chord.

21. The system of claim 20, wherein performing the first operation comprises displaying a first plurality of user interface elements associated with the first chord on the multi-touch surface, and further comprising the steps of receiving a first selection of a first user interface element included in the first plurality of user interface elements, and based on receiving the first selection, executing a first application command associated with the first user interface element.