Zoom-based gesture user interface

• US 8,881,051 B2
• Filed: 07/05/2012
• Issued: 11/04/2014
• Est. Priority Date: 07/05/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Frey, Martin, Hauer, Marcus, Buzzini, Dario, Schaefer, Philipp, Berenson, Adi, Galor, Micha, Metuki, Nili, Shpunt, Alexander
• Primary Examiner(s): PHANTANA ANGKOOL, DAVID
• Application Number:
  US13/541,786
• Publication Number:
  US 20130014052A1
• Time in Patent Office: 852 Days
• Field of Search: 715/800, 715/716, 356/610, 382/154
• US Class Current: 715/800
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

A user interface method, including presenting by a computer executing a user interface, multiple interactive items on a display. A first sequence of images is captured indicating a position in space of a hand of a user in proximity to the display, and responsively to the position, one of the interactive items is associated with the hand. After associating the item, a second sequence of images is captured indicating a movement of the hand, and responsively to the movement, a size of the one of the items is changed on the display.

Claims

1. A user interface method, comprising:

presenting by a computer executing a user interface, multiple interactive items on a display;

capturing a first sequence of images indicating a position in space of a hand of a user in proximity to the display;

responsively to the position, associating a given interactive item with the hand,

wherein associating the given interactive item comprises detecting, in the first sequence of images, a predetermined gesture of the hand, wherein the predetermined gesture comprises closing at least some of the fingers of the hand, and initiating association of the given interactive item with the hand responsively to the gesture;

after associating the item, capturing a second sequence of images indicating a movement of the hand;

responsively to the movement, changing a size of the one of the items on the display; and

releasing the association of the given interactive item from the hand upon detecting that the fingers have opened.

2. The method according to claim 1, wherein the given interactive item is enlarged in response to the movement of the hand comprising motion away from the display.

3. The method according to claim 1, wherein the given interactive item is associated with a function of the computer, and wherein the method comprises activating the function when the given interactive item has been enlarged, in response to the movement, to at least a predetermined threshold size.

4. The method according to claim 3, wherein activating the function comprises playing a media item associated with the given interactive item.

5. The method according to claim 3, wherein activating the function comprises executing a software application.

6. The method according to claim 3, wherein activating the function comprises displaying an image over a full area of the display.

7. The method according to claim 1, wherein associating the given interactive item comprises choosing the interactive item that is presented at a center of the display.

12. The method according to claim 1, wherein at least some of the interactive items contain sub-items, which are not displayed as long as the interactive items presented on the display are smaller than a predetermined size, and wherein enlarging the given interactive item comprises displaying the sub-items contained by the given interactive item when the given interactive item is enlarged to at least the predetermined size.

13. The method according to claim 12, wherein the given interactive item is enlarged in response the movement comprising significant motion away from the display.

14. The method according to claim 1, wherein sensing the position comprises generating a three-dimensional (3D) map of at least a part of body of the user, including the hand.

15. The method according to claim 1, wherein associating the item with the hand comprises identifying the item based on a gaze of the user.

16. The method according to claim 1, wherein associating the item with the hand comprises identifying the item based on a pointing gesture performed the user.

17. The method according to claim 1, wherein presenting the multiple interactive items comprises organizing the multiple interactive items as a grid on the display.

18. The method according to claim 1, wherein changing the size of the one of the items comprises enlarging the one of the items on the display.

8. A user interface method, comprising:

presenting by a computer executing a user interface, multiple interactive items on a display;

capturing a first sequence of images indicating a position in space of a hand of a user in proximity to the display;

responsively to the position, associating a given interactive item with the hand,

wherein associating the given interactive item comprises detecting, in the first sequence of images, a predetermined gesture of the hand, and initiating association of the given interactive item with the hand responsively to the gesture;

after associating the item, capturing a second sequence of images indicating a movement of the hand; and

responsively to the movement, changing a size of the one of the items on the display,

wherein associating the given interactive item comprises detecting, in the first sequence of images, a transverse motion of the hand while the hand maintains the gesture, and associating a different one of the interactive items with the hand responsively to the transverse motion.

9. The method according to claim 8, wherein the predetermined gesture comprises closing at least some of the fingers of the hand.

10. The method according to claim 8, wherein the movement of the hand comprises motion away from the display, and wherein associating the different one of the interactive items with the hand is inhibited when the given interactive item has been enlarged, responsively to the motion, beyond a predetermined threshold size.

11. The method according to claim 10, and comprising assigning less significance to motion along a plane comprising vertical and horizontal axes, upon detecting the motion away from the display.

19. An apparatus, comprising:

a sensing device; and

a computer executing a user interface and configured to present multiple interactive items on a display coupled to the computer, to capture a first sequence of images indicating a position in space of a hand of a user in proximity to the display, to associate, responsively to the position, a given one of the interactive items with the hand, to capture, after associating the given interactive item, a second sequence of images indicating a movement of the hand and to change a size of, responsively to the movement, the given interactive item on the display,

wherein the computer is configured to associate the given interactive item by detecting, in the first sequence of images, a predetermined gesture of the hand, and initiating association of the given interactive item with the hand responsively to the gesture, and to detect, in the first sequence of images, a transverse motion of the hand while the hand maintains the gesture, and to associate a different one of the interactive items with the hand responsively to the transverse motion.

20. The apparatus according to claim 19, wherein the computer is configure to enlarge the given interactive item in response to the movement of the hand comprising motion away from the display.

21. The apparatus according to claim 19, wherein the computer is configured to associated the given interactive item with a function of the computer, and wherein the method comprises activating the function when the given interactive item has been enlarged, in response to the movement, to at least a predetermined threshold size.

22. The apparatus according to claim 21, wherein the computer is configured to activate the function by playing a media item associated with the given interactive item.

23. The apparatus according to claim 21, wherein the computer is configured to activate the function by executing a software application.

24. The apparatus according to claim 21, wherein the computer is configured to activate the function by displaying an image over a full area of the display.

25. The apparatus according to claim 19, wherein the computer is configured to associate the given interactive item by choosing the interactive item that is presented at a center of the display.

26. The apparatus according to claim 19, wherein the predetermined gesture comprises closing at least some of the fingers of the hand.

28. The apparatus according to claim 19, wherein the movement of the hand comprises motion away from the display, and wherein the computer is configured to inhibit associating the different one of the interactive items with the hand when the given interactive item has been enlarged, responsively to the motion, beyond a predetermined threshold size.

29. The apparatus according to claim 28, wherein the computer is configured to assign less significance to motion along a plane comprising vertical and horizontal axes, upon detecting the motion away from the display.

30. The apparatus according to claim 19, wherein at least some of the interactive items contain sub-items, which are not displayed as long as the interactive items presented on the display are smaller than a predetermined size, and wherein the computer is configured to enlarge the given interactive item by displaying the sub-items contained by the given interactive item when the given interactive item is enlarged to at least the predetermined size.

31. The apparatus according to claim 30, wherein the computer is configured to enlarge the given interactive item in response the movement comprising significant motion away from the display.

32. The apparatus according to claim 19, wherein the computer is configured to sense the position by generating a three-dimensional (3D) map of at least a part of body of the user, including the hand.

33. The apparatus according to claim 19, wherein the computer is configured to associate the item with the hand by identifying the item based on a gaze of the user.

34. The apparatus according to claim 19, wherein the computer is configured to associate the item with the hand by identifying the item based on a pointing gesture performed the user.

35. The apparatus according to claim 19, wherein the computer is configured to present the multiple interactive items by organizing the multiple interactive items as a grid on the display.

36. The apparatus according to claim 19, wherein the computer is configured to change the size of the one of the items by enlarging the one of the items on the display.

27. An apparatus, comprising:

a sensing device; and

a computer executing a user interface and configured to present multiple interactive items on a display coupled to the computer, to capture a first sequence of images indicating a position in space of a hand of a user in proximity to the display, to associate, responsively to the position, a given one of the interactive items with the hand, to capture, after associating the given interactive item, a second sequence of images indicating a movement of the hand and to change a size of, responsively to the movement, the given interactive item on the display,

wherein the computer is configured to associate the given interactive item by detecting, in the first sequence of images, a predetermined gesture of the hand comprising closing at least some of the fingers of the hand, and initiating association of the given interactive item with the hand responsively to the gesture, and

to release the association of the given interactive item from the hand upon detecting that the fingers have opened.

37. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a user interface, cause the computer to present multiple interactive items on a display, to capture a first sequence of images indicating a position in space of a hand of a user in proximity to the display, to associate, responsively to the position, a given one of the interactive items with the hand, to capture, after associating the given interactive item, a second sequence of images indicating a movement of the hand, and to change, responsively to the movement, a size of the given interactive item on the display,

wherein the instructions cause the computer to associate the given interactive item by detecting, in the first sequence of images, a predetermined gesture of the hand comprising closing at least some of the fingers of the hand, and initiating association of the given interactive item with the hand responsively to the gesture, and to release the association of the given interactive item from the hand upon detecting that the fingers have opened.

38. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a user interface, cause the computer to present multiple interactive items on a display, to capture a first sequence of images indicating a position in space of a hand of a user in proximity to the display, to associate, responsively to the position, a given one of the interactive items with the hand, to capture, after associating the given interactive item, a second sequence of images indicating a movement of the hand, and to change, responsively to the movement, a size of the given interactive item on the display,

wherein the instructions cause the computer to associate the given interactive item by detecting, in the first sequence of images, a predetermined gesture of the hand, and initiating association of the given interactive item with the hand responsively to the gesture, and to detect, in the first sequence of images, a transverse motion of the hand while the hand maintains the gesture, and to associate a different one of the interactive items with the hand responsively to the transverse motion.