Sessionless pointing user interface

• US 10,642,371 B2
• Filed: 08/21/2018
• Issued: 05/05/2020
• Est. Priority Date: 08/24/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Galor, Micha, Pokrass, Jonathan, Hoffnung, Amir, Or, Ofir
• Primary Examiner(s): Sun, Li P
• Assistant Examiner(s): Wan, Qi
• Application Number:
  US16/106,028
• Publication Number:
  US 20180356898A1
• Time in Patent Office: 623 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method, including receiving, by a computer, a sequence of three-dimensional maps containing at least a hand of a user of the computer, and identifying, in the maps, a device coupled to the computer. The maps are analyzed to detect a gesture performed by the user toward the device, and the device is actuated responsively to the gesture.

Claims

1. A method, comprising:

identifying, by a computer coupled to a three-dimensional (3D) sensing device and a display, different, respective locations of one or more controllable devices other than the display and the computer;

receiving, by the computer from the 3D sensing device, a sequence of three-dimensional maps containing at least a head and a hand of a user of the computer;

detecting in the maps a gaze direction of the user that is directed toward a given device among the one or more controllable devices;

defining a region in space with an apex at the head of the user and a base encompassing the given device;

defining an interaction zone within the region;

analyzing the maps to detect a gesture performed by the hand within the defined interaction zone that is directed toward an identified location of the given device among the one or more controllable devices; and

actuating the given device responsively to the gesture.

2. The method according to claim 1, wherein the gesture is selected from a list comprising a pointing gesture, a grab gesture and a release gesture.

3. The method according to claim 1, and comprising defining an angle threshold, and wherein the gesture comprises extending an elbow associated with the hand at an angle greater than or equal to the angle threshold.

4. The method according to claim 1, and comprising defining a minimum time period, and wherein the gesture comprises extending the hand toward the device and pausing the hand for the minimum time period.

5. The method according to claim 1, and comprising receiving a vocal command from the user, and actuating the device in response to the gesture and the vocal command.

6. The method according to claim 1, and comprising communicating between the computer and a further device upon detecting, in the maps, a subsequent gesture directed toward the further device.

7. The method according to claim 1, wherein the one or more controllable device comprise a plurality of controllable devices, and wherein detecting the gaze direction comprises selecting one of the plurality of the controllable device responsively to the gaze direction.

8. The method according to claim 7, wherein identifying the respective locations comprises performing, by the computer, an initialization step comprising:

identifying, by the computer, the controllable devices that are in proximity to the 3D sensing device; and

directing the user to point to each of the identified controllable devices.

9. An apparatus, comprising:

a display;

a three-dimensional (3D) sensing device; and

a computer coupled to the display and the 3D sensing device and configured to identify different, respective locations of one or more controllable devices other than the display and the computer, to receive from the 3D sensing device a sequence of three-dimensional maps containing at least a head and a hand of a user of the computer, to detect in the maps a gaze direction of the user that is directed toward a given device among the multiple controllable devices, to define a region in space with an apex at the head of the user and a base encompassing the given device, to define an interaction zone within the region, to analyze the maps to detect a gesture performed by the hand within the defined interaction zone that is directed toward an identified location of the given device among the multiple controllable devices, and to actuate the given device responsively to the gesture.

10. The apparatus according to claim 9, wherein the gesture is selected from a list comprising a pointing gesture, a grab gesture and a release gesture.

11. The apparatus according to claim 9, wherein the gesture comprises extending an elbow associated with the hand at an angle greater than or equal to a predefined angle threshold.

12. The apparatus according to claim 9, wherein the gesture comprises extending the hand toward the device and pausing the hand for at least a predefined minimum time period.

13. The apparatus according to claim 9, wherein the computer is configured to receive a vocal command from the user, and to actuate the device in response to the gesture and the vocal command.

14. The apparatus according to claim 9, wherein the computer is configured to communicate with a further device upon detecting, in the maps, a subsequent gesture directed toward the further device.

15. The apparatus according to claim 9, wherein the one or more controllable device comprise a plurality of controllable devices, and wherein the computer is configured to select one of the plurality of the controllable device responsively to the gaze direction.

16. The apparatus according to claim 15, wherein identifying the respective locations comprises performing, by the computer, an initialization step comprising:

identifying, by the computer, the controllable devices that are in proximity to the 3D sensing device; and

directing the user to point to each of the identified controllable devices.

17. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, which is coupled to a display, cause the computer to identify different, respective locations of one or more controllable devices other than the display and the computer, to receive from a 3D sensing device a sequence of three-dimensional maps containing at least a head and a hand of a user of the computer, to detect in the maps a gaze direction of the user that is directed toward a given device among the multiple controllable devices, to define a region in space with an apex at the head of the user and a base encompassing the given device, to define an interaction zone within the region, to analyze the maps to detect a gesture performed by the hand within the defined interaction zone that is directed toward an identified location of the given device among the multiple controllable devices, and to actuate the given device responsively to the gesture.

18. The product according to claim 17, wherein the gesture is selected from a list comprising a pointing gesture, a grab gesture and a release gesture.

19. The product according to claim 17, wherein the gesture comprises extending an elbow associated with the hand at an angle greater than or equal to a predefined angle threshold.

20. The product according to claim 17, wherein the gesture comprises extending the hand toward the device and pausing the hand for at least a predefined minimum time period.