Zoom-based gesture user interface

• US 20130265222A1
• Filed: 05/29/2013
• Published: 10/10/2013
• Est. Priority Date: 07/05/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Berenson, Adi, Galor, Micha, Pokrass, Jonathan, Shani, Ran, Shein, Daniel, Tal, Orlee, Weissenstern, Eran, Frey, Martin, Hoffnung, Amir, Metuki, Nili, Hauer, Marcus, Cohen, Dana, Yaari, Arnon
• Granted Patent: US 9,377,865 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus

Abstract

A method includes arranging, by a computer, multiple interactive objects as a hierarchical data structure, each node of the hierarchical data structure associated with a respective one of the multiple interactive objects, and presenting, on a display coupled to the computer, a first subset of the multiple interactive objects that are associated with one or more child nodes of one of the multiple interactive objects. A sequence of three-dimensional (3D) maps including at least part of a hand of a user positioned in proximity to the display is received, and the hand performing a transverse gesture followed by a grab gesture followed by a longitudinal gesture followed by an execute gesture is identified in the sequence of three-dimensional (3D) maps, and an operation associated with the selected object is accordingly performed.

Claims

1. A method, comprising:

arranging, by a computer, multiple interactive objects as a hierarchical data structure, each node of the hierarchical data structure associated with a respective one of the multiple interactive objects;

presenting, on a display coupled to the computer, a first subset of the multiple interactive objects that are associated with one or more child nodes of one of the multiple interactive objects;

receiving a sequence of three-dimensional (3D) maps including at least part of a hand of a user positioned in proximity to the display;

identifying, in the sequence of three-dimensional (3D) maps, the hand performing a transverse gesture followed by a grab gesture followed by a longitudinal gesture followed by an execute gesture;

selecting, in response to the transverse gesture, an object from the first subset of the multiple interactive objects; and

performing, in response to the grab gesture, the longitudinal gesture and the execute gesture, an operation associated with the selected object.

2. The method according to claim 1, wherein the execute gesture is selected from a list comprising a timeclick gesture, a release gesture and a subsequent transverse gesture.

3. The method according to claim 1, wherein the longitudinal gesture comprises a pull gesture, wherein the selected one of the first subset of the multiple interactive objects comprises a media file, and wherein the operation comprises playing the media file.

4. The method according to claim 3, and comprising presenting context information on the media file upon detecting the pull gesture.

5. The method according to claim 3, and comprising receiving, while playing the media file, an additional set of 3D maps, detecting, in the additional set of 3D maps, an additional grab gesture followed by an additional transverse gesture followed by a further grab gesture followed by an additional release gesture, presenting one or more media controls in response to the additional grab gesture, positioning a cursor in response to the transverse gesture in the additional set of 3D maps, identifying, upon detecting the further grab gesture, one of the one or more media controls presented in proximity to the cursor, and performing an operation associated with the one of the one or more media controls upon detecting the additional release gesture

6. The method according to claim 5, wherein each of the one or more media controls is selected from a list comprising a play and pause control, a seek control, a volume and mute control, a fast forward control, a rewind control, a next track control and a previous track control.

7. The method according to claim 1, wherein the longitudinal gesture comprises a pull gesture, wherein the selected one of the first subset of the multiple interactive objects comprises a software application, and wherein the operation comprises executing the software application.

8. The method according to claim 7, and comprising presenting context information on the software application upon detecting the pull gesture.

9. The method according to claim 1, wherein the longitudinal gesture comprises a pull gesture, and wherein the operation comprises presenting, on the display, a second subset of the multiple interactive objects that are associated with one or more child nodes of the selected one of the first subset of the multiple interactive objects.

10. The method according to claim 9, and comprising presenting a preview of the second subset of the multiple interactive objects upon detecting the pull gesture.

11. The method according to claim 1, wherein the longitudinal gesture comprises a push gesture, and wherein the operation comprises presenting, on the display, a second subset of the multiple interactive objects that are associated with one or more child nodes of a parent of the selected one of the first subset of the multiple interactive objects.

12. The method according to claim 11, and comprising presenting a preview of the second subset of the multiple interactive objects upon detecting the push gesture.

13. The method according to claim 1, and comprising assigning less significance to a transverse motion of the hand than to a longitudinal motion of the hand upon detecting a start of the longitudinal gesture.

14. The method according to claim 1, and comprising assigning less significance to a longitudinal motion of the hand than to a transverse motion of the hand upon detecting a start of the transverse gesture.

15. The method according to claim 1, and comprising changing a layout of the subset of the multiple interactive objects in response to the longitudinal gesture.

16. An apparatus, comprising:

a sensing device;

a display; and

a computer coupled to the sensing device and the display, and configured to arrange multiple interactive objects as a hierarchical data structure, each node of the hierarchical data structure associated with a respective one of the multiple interactive objects, to present, on the display, a first subset of the multiple interactive objects that are associated with one or more child nodes of one of the multiple interactive objects, to receive a sequence of three-dimensional (3D) maps including at least part of a hand of a user positioned in proximity to a display coupled to the computer, to identify, in the sequence of three-dimensional (3D) maps, the hand performing a transverse gesture followed by a grab gesture followed by a longitudinal gesture followed by an execute gesture, to select, in response to the transverse gesture, one of the first subset of the multiple interactive objects, and to perform, in response to the grab gesture, the longitudinal and the execute gesture, an operation associated with the selected one of the first subset of the multiple interactive objects.

17. The apparatus according to claim 16, wherein the execute gesture is selected from a list comprising a timeclick gesture, a release gesture and a subsequent transverse gesture.

18. The apparatus according to claim 16, wherein the longitudinal gesture comprises a pull gesture, wherein the selected one of the first subset of the multiple interactive objects comprises a media file, and wherein the operation comprises playing the media file.

19. The apparatus according to claim 18, wherein the computer is configured to present context information on the media file upon detecting the pull gesture.

20. The apparatus according to claim 18, wherein the computer is configured to receive, while playing the media file, an additional set of 3D maps, to detect, in the additional set of 3D maps, an additional grab gesture followed by an additional transverse gesture followed by a further grab gesture followed by an additional release gesture, to present one or more media controls in response to the additional grab gesture, to position a cursor in response to the additional transverse gesture, to identify, upon detecting the further grab gesture, one of the one or more media controls presented in proximity to the cursor, and to perform an operation associated with the one of the one or more media controls upon detecting the additional release gesture.

21. The apparatus according to claim 20, wherein each of the one or more media controls is selected from a list comprising a play and pause control, a seek control, a volume and mute control, a fast forward control, a rewind control, a next track control and a previous track control.

22. The apparatus according to claim 16, wherein the longitudinal gesture comprises a pull gesture, wherein the selected one of the first subset of the multiple interactive objects comprises a software application, and wherein the operation comprises executing the software application.

23. The apparatus according to claim 22, wherein the computer is configured to present context information on the software application upon detecting the pull gesture.

24. The apparatus according to claim 16, wherein the longitudinal gesture comprises a pull gesture, and wherein the operation comprises presenting, on the display, a second subset of the multiple interactive objects that are associated with one or more child nodes of the selected one of the first subset of the multiple interactive objects.

25. The apparatus according to claim 24, wherein the computer is configured to present a preview of the second subset of the multiple interactive objects upon detecting the pull gesture.

26. The apparatus according to claim 16, wherein the longitudinal gesture comprises a push gesture, and wherein the operation comprises presenting, on the display, a second subset of the multiple interactive objects that are associated with one or more child nodes of a parent of the selected one of the first subset of the multiple interactive objects.

27. The apparatus according to claim 26, wherein the computer is configured to present a preview of the second subset of the multiple interactive objects upon detecting the push gesture.

28. The apparatus according to claim 16, wherein the computer is configured to assign less significance to a transverse motion of the hand than to a longitudinal motion of the hand upon detecting a start of the longitudinal gesture.

29. The apparatus according to claim 16, wherein the computer is configured to assign less significance to a longitudinal motion of the hand than to a transverse motion of the hand upon detecting a start of the transverse gesture.

30. The apparatus according to claim 16, wherein the computer is configured to change a layout of the subset of the multiple interactive objects in response to the longitudinal gesture.

31. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a user interface, cause the computer to arrange multiple interactive objects as a hierarchical data structure, each node of the hierarchical data structure associated with a respective one of the multiple interactive objects, to present, on a display, a first subset of the multiple interactive objects that are associated with one or more child nodes of one of the multiple interactive objects, to receive a sequence of three-dimensional (3D) maps including at least part of a hand of a user positioned in proximity to a display coupled to the computer, to identify, in the sequence of three-dimensional (3D) maps, the hand performing a transverse gesture followed by a grab gesture followed by a longitudinal gesture followed by an execute gesture, to select, in response to the transverse gesture, one of the first subset of the multiple interactive objects, and to perform, in response to the execute gesture, the longitudinal and the execute gesture, an operation associated with the selected one of the first subset of the multiple interactive objects.