Zoom-based gesture user interface

• US 9,377,865 B2
• Filed: 05/29/2013
• Issued: 06/28/2016
• Est. Priority Date: 07/05/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Berenson, Adi, Cohen, Dana, Galor, Micha, Pokrass, Jonathan, Shani, Ran, Shein, Daniel, Tal, Orlee, Yaari, Arnon, Weissenstern, Eran, Frey, Martin, Hoffnung, Amir, Metuki, Nili, Hauer, Marcus
• Primary Examiner(s): Angkool, David Phantana
• Application Number:
  US13/904,050
• Publication Number:
  US 20130265222A1
• Time in Patent Office: 1,126 Days
• Field of Search: 715/800
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus

Abstract

A method includes arranging, by a computer, multiple interactive objects as a hierarchical data structure, each node of the hierarchical data structure associated with a respective one of the multiple interactive objects, and presenting, on a display coupled to the computer, a first subset of the multiple interactive objects that are associated with one or more child nodes of one of the multiple interactive objects. A sequence of three-dimensional (3D) maps including at least part of a hand of a user positioned in proximity to the display is received, and the hand performing a transverse gesture followed by a grab gesture followed by a longitudinal gesture followed by an execute gesture is identified in the sequence of three-dimensional (3D) maps, and an operation associated with the selected object is accordingly performed.

Claims

1. A method, comprising:

arranging, by a computer, multiple interactive objects as a hierarchical data structure, each node of the hierarchical data structure associated with a respective one of the multiple interactive objects;

presenting, on a display coupled to the computer, a first subset of the multiple interactive objects that are associated with one or more child nodes of one of the multiple interactive objects;

receiving a sequence of three-dimensional (3D) maps including at least part of a hand of a user positioned in proximity to the display;

identifying, in the sequence of three-dimensional (3D) maps, the hand performing a transverse gesture followed by a grab gesture followed by a longitudinal gesture followed by an execute gesture;

selecting, in response to the transverse gesture, an object from the first subset of the multiple interactive objects; and

in response to the grab gesture, the longitudinal gesture and the execute gesture, presenting, on the display, a second subset of the multiple interactive objects that are associated with one or more child nodes of the selected object,

wherein the execute gesture is selected from a list comprising a timeclick gesture, a release gesture and a subsequent transverse gesture.

2. The method according to claim 1, wherein the longitudinal gesture comprises a pull gesture.

3. The method according to claim 2, and comprising presenting a preview of the second subset of the multiple interactive objects upon detecting the pull gesture.

4. The method according to claim 1, and comprising assigning less significance to a transverse motion of the hand than to a longitudinal motion of the hand upon detecting a start of the longitudinal gesture.

5. The method according to claim 1, and comprising assigning less significance to a longitudinal motion of the hand than to a transverse motion of the hand upon detecting a start of the transverse gesture.

6. The method according to claim 1, and comprising changing a layout of the subset of the multiple interactive objects in response to the longitudinal gesture.

7. A method, comprising:

arranging, by a computer, multiple interactive objects as a hierarchical data structure, each node of the hierarchical data structure associated with a respective one of the multiple interactive objects;

presenting, on a display coupled to the computer, a subset of the multiple interactive objects that are associated with one or more child nodes of one of the multiple interactive objects;

receiving a sequence of three-dimensional (3D) maps including at least part of a hand of a user positioned in proximity to the display;

identifying, in the sequence of three-dimensional (3D) maps, the hand performing a transverse gesture followed by a grab gesture followed by a pull gesture followed by an execute gesture;

selecting, in response to the transverse gesture, a media file from the subset of the multiple interactive objects;

in response to the grab gesture, the longitudinal gesture and the execute gesture, playing the selected media file; and

receiving, while playing the media file, an additional set of 3D maps, detecting, in the additional set of 3D maps, an additional grab gesture followed by an additional transverse gesture followed by a further grab gesture followed by an additional release gesture, presenting one or more media controls in response to the additional grab gesture, positioning a cursor in response to the transverse gesture in the additional set of 3D maps, identifying, upon detecting the further grab gesture, one of the one or more media controls presented in proximity to the cursor, and performing an operation associated with the one of the one or more media controls upon detecting the additional release gesture.

8. The method according to claim 7, wherein each of the one or more media controls is selected from a list comprising a play and pause control, a seek control, a volume and mute control, a fast forward control, a rewind control, a next track control and a previous track control.

9. An apparatus, comprising:

a sensing device;

a display; and

a computer coupled to the sensing device and the display, and configured to arrange multiple interactive objects as a hierarchical data structure, each node of the hierarchical data structure associated with a respective one of the multiple interactive objects, to present, on the display, a first subset of the multiple interactive objects that are associated with one or more child nodes of one of the multiple interactive objects, to receive a sequence of three-dimensional (3D) maps including at least part of a hand of a user positioned in proximity to a display coupled to the computer, to identify, in the sequence of three-dimensional (3D) maps, the hand performing a transverse gesture followed by a grab gesture followed by a longitudinal gesture followed by an execute gesture, to select, in response to the transverse gesture, one of the first subset of the multiple interactive objects, and, in response to the grab gesture, the longitudinal gesture and the execute gesture, to present, on the display, a second subset of the multiple interactive objects that are associated with one or more child nodes of the selected one of the first subset of the multiple interactive objects,

wherein the execute gesture is selected from a list comprising a timeclick gesture, a release gesture and a subsequent transverse gesture.

10. The apparatus according to claim 9, wherein the longitudinal gesture comprises a pull gesture.

11. The apparatus according to claim 10, wherein the computer is configured to present a preview of the second subset of the multiple interactive objects upon detecting the pull gesture.

12. The apparatus according to claim 9, wherein the computer is configured to assign less significance to a transverse motion of the hand than to a longitudinal motion of the hand upon detecting a start of the longitudinal gesture.

13. The apparatus according to claim 9, wherein the computer is configured to assign less significance to a longitudinal motion of the hand than to a transverse motion of the hand upon detecting a start of the transverse gesture.

14. The apparatus according to claim 9, wherein the computer is configured to change a layout of the subset of the multiple interactive objects in response to the longitudinal gesture.

15. An apparatus, comprising:

a sensing device;

a display; and

a computer coupled to the sensing device and the display, and configured;

to arrange multiple interactive objects as a hierarchical data structure, each node of the hierarchical data structure associated with a respective one of the multiple interactive objects,

to present, on the display, a first subset of the multiple interactive objects that are associated with one or more child nodes of one of the multiple interactive objects,

to receive a sequence of three-dimensional (3D) maps including at least part of a hand of a user positioned in proximity to a display coupled to the computer,

to identify, in the sequence of three-dimensional (3D) maps, the hand performing a transverse gesture followed by a grab gesture followed by a pull gesture followed by an execute gesture,

to select, in response to the transverse gesture, a media file from the subset of the multiple interactive objects,

to perform, in response to the grab gesture, the pull gesture and the execute gesture, an operation associated with the selected one of the first subset of the multiple interactive objects,

to receive, while playing the media file, an additional set of 3D maps, to detect, in the additional set of 3D maps, an additional grab gesture followed by an additional transverse gesture followed by a further grab gesture followed by an additional release gesture, to present one or more media controls in response to the additional grab gesture, to position a cursor in response to the additional transverse gesture, to identify, upon detecting the further grab gesture, one of the one or more media controls presented in proximity to the cursor, and to perform an operation associated with the one of the one or more media controls upon detecting the additional release gesture.

16. The apparatus according to claim 15, wherein each of the one or more media controls is selected from a list comprising a play and pause control, a seek control, a volume and mute control, a fast forward control, a rewind control, a next track control and a previous track control.

17. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a user interface, cause the computer to arrange multiple interactive objects as a hierarchical data structure, each node of the hierarchical data structure associated with a respective one of the multiple interactive objects, to present, on a display, a first subset of the multiple interactive objects that are associated with one or more child nodes of one of the multiple interactive objects, to receive a sequence of three-dimensional (3D) maps including at least part of a hand of a user positioned in proximity to a display coupled to the computer, to identify, in the sequence of three-dimensional (3D) maps, the hand performing a transverse gesture followed by a grab gesture followed by a longitudinal gesture followed by an execute gesture, to select, in response to the transverse gesture, one of the first subset of the multiple interactive objects, and, in response to the execute gesture, the longitudinal gesture and the execute gesture, to present, on the display, a second subset of the multiple interactive objects that are associated with one or more child nodes of the selected one of the first subset of the multiple interactive objects,

wherein the execute gesture is selected from a list comprising a timeclick gesture, a release gesture and a subsequent transverse gesture.