Disambiguation of multitouch gesture recognition for 3D interaction

• US 9,529,440 B2
• Filed: 07/15/2013
• Issued: 12/27/2016
• Est. Priority Date: 01/25/1999
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Piemonte, Patrick S., Moore, Bradford A., Chen, Billy P.
• Primary Examiner(s): TANK, ANDREW L
• Application Number:
  US13/942,369
• Publication Number:
  US 20140019917A1
• Time in Patent Office: 1,261 Days
• Field of Search: 715/849, 715/863, 715/964
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images

Abstract

A multitouch device can interpret and disambiguate different gestures related to manipulating a displayed image of a 3D object, scene, or region. Examples of manipulations include pan, zoom, rotation, and tilt. The device can define a number of manipulation modes, including one or more single-control modes such as a pan mode, a zoom mode, a rotate mode, and/or a tilt mode. The manipulation modes can also include one or more multi-control modes, such as a pan/zoom/rotate mode that allows multiple parameters to be modified simultaneously.

Claims

1. A method comprising:

at an electronic device with a display and a touch-sensitive surface;

displaying, on the display, an image of a 3D region;

detecting an initial portion of an input on the touch-sensitive surface, the initial portion of the input having characteristics including;

one or more contacts; and

an initial motion of at least one of the one or more contacts;

while continuing to detect the input and prior to detecting a termination of the input, wherein termination of the input is detected based on liftoff of the one or more contacts included in the input;

determining, based on the characteristics of the initial portion of the input, a manipulation mode, wherein;

the manipulation mode is determined from a plurality of modes including at least one single-control mode and at least one multi-control mode;

a respective single-control mode allows a single first type of modification to the image of the 3D region at a time during a remainder of the input, and a respective multi-control mode allows two or more types of modifications of the image of the 3D region simultaneously during the remainder of the input including a second type of modification of the image of the 3D region that is different from the first type of modification of the image of the 3D region;

in accordance with a determination, based on the characteristics of the initial portion of the input, that the input has been recognized as a first gesture;

entering a first single-control mode that is associated specifically with the first gesture, and

modifying the image of the 3D region in the first single-control mode until the input terminates including preventing the second type of modification of the image of the 3D region until liftoff of the one or more contacts included in the input has been detected; and

,

in accordance with a determination, based on the characteristics of the initial portion of the input, that the input has not been recognized as any single gesture;

entering a first multi-control mode that is associated with a first plurality of different gestures, and

modifying the image of the 3D region in the first multi-control mode until the input terminates including permitting modification of the image of the 3D region using the two or more types of modifications of the image of the 3D region including the second type of modification of the image of the 3D region until liftoff of the one or more contacts included in the input has been detected.

2. The method of claim 1, wherein the plurality of modes includes:

a single-control mode wherein the image is modified by modifying a tilt angle of the image relative to a base plane;

a single-control mode wherein the image is modified by laterally shifting a field of view across a viewing plane;

a single-control mode wherein the image is modified by enlarging or shrinking the image;

a single-control mode wherein the image is modified by rotating the field of view in the viewing plane; and

a multi-control mode wherein the image is modifiable by a combination of the modifications associated with the tilt mode, the pan mode, and the zoom mode.

3. The method of claim 1, wherein the image of the 3D region comprises a map or a CAD object.

4. The method of claim 1, wherein:

determining the manipulation mode includes comparing the initial portion of the input to respective criteria for gestures, in a second plurality of different gestures, that are associated with corresponding single-control modes; and

entering the first multi-control mode includes selecting two or more of the gestures in the second plurality of different gestures that most closely match the initial portion of the input, while excluding one or more gestures in the second plurality of different gestures that do not match the initial portion of the input as closely.

5. The method of claim 1, wherein the plurality of modes includes:

a single-control mode that only allows zooming of the image of the 3D region during the remainder of the input;

a single-control mode that only allows rotation of the 3D region during the remainder of the input; and

a multi-control mode that allows simultaneous zooming and rotation of the 3D region during the remainder of the input.

6. An electronic device, comprising:

a display;

touch-sensitive surface;

one or more processors;

memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

displaying, on the display, an image of a 3D region;

detecting an initial portion of an input on the touch-sensitive surface, the initial portion of the input having characteristics including;

one or more contacts; and

an initial motion of at least one of the one or more contacts;

while continuing to detect the input and prior to detecting a termination of the input, wherein termination of the input is detected based on liftoff of the one or more contacts included in the input;

determining, based on the characteristics of the initial portion of the input, a manipulation mode, wherein;

the manipulation mode is determined from a plurality of modes including at least one single-control mode and at least one multi-control mode;

a respective single-control mode allows a single first type of modification to the image of the 3D region at a time during a remainder of the input, and

a respective multi-control mode allows two or more types of modifications of the image of the 3D region simultaneously during the remainder of the input including a second type of modification of the image of the 3D region that is different from the first type of modification of the image of the 3D region;

in accordance with a determination, based on the characteristics of the initial portion of the input, that the input has been recognized as a first gesture;

entering a first single-control mode that is associated specifically with the first gesture, and

modifying the image of the 3D region in the first single-control mode until the input terminates including preventing the second type of modification of the image of the 3D region until liftoff of the one or more contacts included in the input has been detected; and

,

in accordance with a determination, based on the characteristics of the initial portion of the input, that the input has not been recognized as any single gesture;

entering a first multi-control mode that is associated with a first plurality of different gestures, and

modifying the image of the 3D region in the first multi-control mode until the input terminates including permitting modification of the image of the 3D region using the two or more types of modifications of the image of the 3D region including the second type of modification of the image of the 3D region until liftoff of the one or more contacts included in the input has been detected.

7. The device of claim 6, wherein the plurality of modes includes:

a single-control mode wherein the image is modified by modifying a tilt angle of the image relative to a base plane;

a single-control mode wherein the image is modified by laterally shifting a field of view across a viewing plane;

a single-control mode wherein the image is modified by enlarging or shrinking the image;

a single-control mode wherein the image is modified by rotating the field of view in the viewing plane; and

a multi-control mode wherein the image is modifiable by a combination of the modifications associated with the tilt mode, the pan mode, and the zoom mode.

8. The device of claim 6, wherein the image of the 3D region comprises a map or a CAD object.

9. The device of claim 6, wherein:

determining the manipulation mode includes comparing the initial portion of the input to respective criteria for gestures, in a second plurality of different gestures, that are associated with corresponding single-control modes; and

entering the first multi-control mode includes selecting two or more of the gestures in the second plurality of different gestures that most closely match the initial portion of the input, while excluding one or more gestures in the second plurality of different gestures that do not match the initial portion of the input as closely.

10. The device of claim 6, wherein the plurality of modes includes:

a single-control mode that only allows zooming of the image of the 3D region during the remainder of the input;

a single-control mode that only allows rotation of the 3D region during the remainder of the input; and

a multi-control mode that allows simultaneous zooming and rotation of the 3D region during the remainder of the input.

11. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device with a display and a touch-sensitive surface, cause the electronic device to:

display, on the display, an image of a 3D region;

detect an initial portion of an input on the touch-sensitive surface, the initial portion of the input having characteristics including;

one or more contacts; and

an initial motion of at least one of the one or more contacts;

while continuing to detect the input and prior to detecting a termination of the input, wherein termination of the input is detected based on liftoff of the one or more contacts included in the input;

determine, based on the characteristics of the initial portion of the input, a manipulation mode, wherein;

the manipulation mode is determined from a plurality of modes including at least one single-control mode and at least one multi-control mode;

a respective single-control mode allows a single first type of modification to the image of the 3D region at a time during a remainder of the input, and

a respective multi-control mode allows two or more types of modifications of the image of the 3D region simultaneously during the remainder of the input including a second type of modification of the image of the 3D region that is different from the first type of modification of the image of the 3D region;

in accordance with a determination, based on the characteristics of the initial portion of the input, that the input has been recognized as a first gesture;

enter a first single-control mode that is associated specifically with the first gesture, and

modify the image of the 3D region in the first single-control mode until the input terminates including preventing the second type of modification of the image of the 3D region until liftoff of the one or more contacts included in the input has been detected; and

,

in accordance with a determination, based on the characteristics of the initial portion of the input, that the input has not been recognized as any single gesture;

enter a first multi-control mode that is associated with a first plurality of different gestures, and

modify the image of the 3D region in the first multi-control mode until the input terminates including permitting modification of the image of the 3D region using the two or more types of modifications of the image of the 3D region including the second type of modification of the image of the 3D region until liftoff of the one or more contacts included in the input has been detected.

12. The computer readable storage medium of claim 11, wherein the plurality of modes includes:

a single-control mode wherein the image is modified by modifying a tilt angle of the image relative to a base plane;

a single-control mode wherein the image is modified by laterally shifting a field of view across a viewing plane;

a single-control mode wherein the image is modified by enlarging or shrinking the image;

a single-control mode wherein the image is modified by rotating the field of view in the viewing plane; and

a multi-control mode wherein the image is modifiable by a combination of the modifications associated with the tilt mode, the pan mode, and the zoom mode.

13. The computer readable storage medium of claim 11, wherein the image of the 3D region comprises a map or a CAD object.

14. The computer readable storage medium of claim 11, wherein:

determining the manipulation mode includes comparing the initial portion of the input to respective criteria for gestures, in a second plurality of different gestures, that are associated with corresponding single-control modes; and

entering the first multi-control mode includes selecting two or more of the gestures in the second plurality of different gestures that most closely match the initial portion of the input, while excluding one or more gestures in the second plurality of different gestures that do not match the initial portion of the input as closely.

15. The computer readable storage medium of claim 11, wherein the plurality of modes includes:

a single-control mode that only allows zooming of the image of the 3D region during the remainder of the input;

a single-control mode that only allows rotation of the 3D region during the remainder of the input; and

a multi-control mode that allows simultaneous zooming and rotation of the 3D region during the remainder of the input.