POINTING-BASED DISPLAY INTERACTION

• US 20130321271A1
• Filed: 08/07/2013
• Published: 12/05/2013
• Est. Priority Date: 02/09/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Bychkov, Eyal, Brezner, Oren, Galor, Micha, Or, Ofir, Pokrass, Jonathan, Hoffnung, Amir, Berliner, Tamir
• Granted Patent: US 9,342,146 B2
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A method includes receiving and segmenting a first sequence of three-dimensional (3D) maps over time of at least a part of a body of a user of a computerized system in order to extract 3D coordinates of a first point and a second point of the user, the 3D maps indicating a motion of the second point with respect to a display coupled to the computerized system. A line segment that intersects the first point and the second point is calculated, and a target point is identified where the line segment intersects the display. An interactive item presented on the display in proximity to the target point is engaged.

Claims

1. A method, comprising:

receiving and segmenting a first sequence of three-dimensional (3D) maps over time of at least a part of a body of a user of a computerized system in order to extract 3D coordinates of a first point and a second point of the user, the 3D maps indicating a motion of the second point with respect to a display coupled to the computerized system;

calculating a line segment that intersects the first point and the second point;

identifying a target point where the line segment intersects the display; and

engaging an interactive item presented on the display in proximity to the target point.

2. The method according to claim 1, wherein the first point is selected from a list comprising an eye of the user, a point between eyes of the user

3. The method according to claim 1, wherein the first point comprises a knuckle of the finger.

4. The method according to claim 1, wherein the second point comprises a limb of the user.

5. The method according to claim 1 wherein the second point comprises a tip of a finger.

6. The method according to claim 1, wherein engaging the interactive item comprises performing an operation associated with the interactive item upon receiving a vocal command from the user.

7. The method according to claim 1, wherein the motion with respect to the display comprises a motion toward the display;

8. The method according to claim 7, wherein engaging the interactive item comprises performing an action associated with the interactive item upon receiving a second sequence of 3D maps indicating a deceleration of the motion of the second point toward the display, and receiving a third sequence of 3D maps indicating a motion of the second point away from the display.

9. The method according to claim 7, wherein engaging the interactive item comprises repositioning the interactive item on the display responsively upon receiving a second sequence of 3D maps indicating a deceleration of the motion of the second point toward the display, and receiving a third sequence of 3D maps indicating a motion of the second point along a plane comprising vertical and horizontal axes.

10. The method according to claim 7, wherein engaging the interactive item comprises performing a hold operation on the interactive item responsively upon receiving a second sequence of 3D maps indicating a deceleration of the motion of the second point toward the display, and receiving a third sequence of 3D maps indicating that the user is holding the second point relatively steady for a specific time period.

11. The method according to claim 10, wherein the hold operation comprises presenting context information for the interactive item.

12. The method according to claim 10, wherein the hold operation comprises deleting the interactive item from the display.

13. The method according to claim 7, wherein the second point comprises a tip of an index finger, and wherein engaging the interactive item comprises selecting the interactive item upon receiving a second sequence of 3D maps indicating a deceleration of the motion of the second point toward the display, and receiving a third sequence of 3D maps indicating the user folding a thumb toward the index finger.

14. The method according to claim 1, and comprising calculating a calibration coefficient based on the proximity of the interactive item to the target point.

15. The method according to claim 1, and comprising identifying the line segment intersecting a device coupled to the computer, and controlling a function of the device responsively to a motion of the second point.

16. An apparatus, comprising:

a display; and

a computer coupled to the display and configured to receive and segment a first sequence of three-dimensional (3D) maps over time of at least a part of a body of a user of a computerized system in order to extract 3D coordinates of a first point and a second point of the user, the 3D maps indicating a motion of the second point with respect to a display coupled to the computer system, to calculate a line segment that intersects the first point and the second point, to identify a target point where the line segment intersects the display, and to engage an interactive item presented on the display in proximity to the target point.

17. The apparatus according to claim 16, wherein the computer is configured to select the first point from a list comprising an eye of the user, a point between eyes of the user

18. The apparatus according to claim 16, wherein the first point comprises a knuckle of the finger.

19. The apparatus according to claim 16, wherein the second point comprises a limb of the user.

20. The apparatus according to claim 16 wherein the second point comprises a tip of a finger.

21. The apparatus according to claim 16, wherein the computer is configured to engage the interactive item by performing an operation associated with the interactive item upon receiving a vocal command from the user.

22. The apparatus according to claim 16, wherein the motion with respect to the display comprises a motion toward the display;

23. The apparatus according to claim 22, wherein the computer is configured to engage the interactive item by performing an action associated with the interactive item upon receiving a second sequence of 3D maps indicating a deceleration of the motion of the second point toward the display, and receiving a third sequence of 3D maps indicating a motion of the second point away from the display.

24. The apparatus according to claim 22, wherein the computer is configured to engage the interactive item by repositioning the interactive item on the display responsively upon receiving a second sequence of 3D maps indicating a deceleration of the motion of the second point toward the display, and receiving a third sequence of 3D maps indicating a motion of the second point along a plane comprising vertical and horizontal axes.

25. The apparatus according to claim 22, wherein the computer is configured to engage the interactive item by performing a hold operation on the interactive item responsively upon receiving a second sequence of 3D maps indicating a deceleration of the motion of the second point toward the display, and receiving a third sequence of 3D maps indicating that the user is holding the second point relatively steady for a specific time period.

26. The apparatus according to claim 25, wherein the computer is configured to perform the hold operation by presenting context information for the interactive item.

27. The apparatus according to claim 25, wherein the computer is configured to perform the hold operation by deleting the interactive item from the display.

28. The apparatus according to claim 22, wherein the second point comprises a tip of an index finger, and wherein the computer is configured to engage the interactive item by selecting the interactive item upon receiving a second sequence of 3D maps indicating a deceleration of the motion of the second point toward the display, and receiving a third sequence of 3D maps indicating the user folding a thumb toward the index finger.

29. The apparatus according to claim 16, wherein the computer is configured to calculate a calibration coefficient based on the proximity of the interactive item to the target point.

30. The apparatus according to claim 16, wherein the computer is configured to identify the line segment intersecting a device coupled to the computer, and to control a function of the device responsively to a motion of the second point.

31. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive and segment a first sequence of three-dimensional (3D) maps over time of at least a part of a body of a user of the computer in order to extract 3D coordinates of a first point and a second point of the user, the 3D maps indicating a motion of the second point with respect to a display coupled to the computer, to calculate a line segment that intersects the first point and the second point, to identify a target point where the line segment intersects the display, and to engage an interactive item presented on the display in proximity to the target point.