Sessionless pointing user interface

• US 10,088,909 B2
• Filed: 10/22/2015
• Issued: 10/02/2018
• Est. Priority Date: 08/24/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Galor, Micha, Pokrass, Jonathan, Hoffnung, Amir, Or, Ofir
• Primary Examiner(s): To, Jennifer
• Assistant Examiner(s): Wan, Jasmine
• Application Number:
  US14/919,751
• Publication Number:
  US 20160041623A1
• Time in Patent Office: 1,076 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method, including receiving, by a computer, a sequence of three-dimensional maps containing at least a hand of a user of the computer, and identifying, in the maps, a device coupled to the computer. The maps are analyzed to detect a gesture performed by the user toward the device, and the device is actuated responsively to the gesture.

Claims

1. A method, the method comprising:

identifying, by a computer with a display, a controlled device that is controlled by the computer, the controlled device is other than the computer display;

receiving, by a computer, a sequence of three-dimensional maps including at least an arm, an elbow and a hand, of a user of the computer;

detecting, in the maps, a gaze direction of the user;

analyzing the maps to detect a pointing gesture performed by the arm and the hand of the user toward the controlled device,

wherein analyzing the maps comprises defining a pyramid shaped region having an apex meeting the user and a base encompassing the device, and defining an interaction region contained within the pyramid shaped region; and

actuating the controlled device responsively to the pointing gesture on condition that the elbow is extended in the pointing gesture toward the controlled device at an angle no less than a predefined angular threshold, and on condition that the hand is positioned within the interaction region and moved within the interaction region toward the controlled device, and the gaze direction.

2. The method according to claim 1, wherein the controlled device is actuated only when the elbow has been extended in the gesture for at least a predefined minimum time period.

3. The method according to claim 1, and comprising receiving a vocal command from the user, wherein the controlled device is actuated in response to the gesture and the vocal command.

4. The method according to claim 1, and comprising detecting, in the maps, a gaze direction of the user, wherein the controlled device is actuated in response to the gesture and the gaze direction.

5. The method according to claim 1, and comprising communicating between the controlled device and a further device upon detecting, in the maps, a subsequent gesture directed toward the further device.

6. The method according to claim 1, wherein the predefined angular threshold is at least 90°

, such that the controlled device is actuated only when the elbow is extended in the pointing gesture toward the controlled device at an angle greater than or equal to 90°

.

7. An apparatus, comprising:

a three-dimensional sensing device; and

a computer configured to identify a controlled device that is controlled by the computer, the controlled device is other than a computer display, to receive from the three-dimensional sensing device a sequence of three-dimensional maps including at least an arm, an elbow and a hand, of a user of the computer, to detect, in the maps, a gaze direction of the user, to analyze the maps to detect a pointing gesture performed by the arm and the hand of the user toward the controlled device and to define a pyramid shaped region having an apex that meets the user and a base encompassing the device and an interaction region contained within the pyramid shaped region, and to actuate the controlled device responsively to the pointing gesture on condition that the elbow is extended in the pointing gesture toward the controlled device at an angle no less than a predefined angular threshold, and on condition that the hand is positioned within the interaction region and is moved within the interaction region toward the controlled device and the gaze direction.

8. The apparatus according to claim 7, wherein the computer is configured to actuate the controlled device only when the elbow has been extended in the gesture for at least a predefined minimum time period.

9. The apparatus according to claim 7, wherein the computer is configured to receive a vocal command from the user, and to actuate the controlled device in response to the gesture and the vocal command.

10. The apparatus according to claim 7, wherein the computer is configured to detect, in the maps, a gaze direction of the user, and to actuate the controlled device in response to the gesture and the gaze direction.

11. The apparatus according to claim 7, wherein the computer is configured to convey communication between the controlled device and a further device upon detecting, in the maps, a subsequent gesture directed toward the further device.

12. The apparatus according to claim 7, wherein the predefined angular threshold is at least 90°

, such that the controlled device is actuated only when the elbow is extended in the pointing gesture toward the controlled device at an angle greater than or equal to 90°

.

13. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to identify a controlled device that is controlled by the computer, the controlled device is other than a computer display, to receive a sequence of three-dimensional maps including at least an arm, an elbow and a hand, of a user of the computer, to detect, in the maps, a gaze direction of the user, to analyze the maps to detect a pointing gesture performed by the arm and the hand of the user toward the controlled device and to define a pyramid shaped region having an apex that meets the user and a base encompassing the device and an interaction region contained within the pyramid shaped region, and to actuate the controlled device responsively to the pointing gesture on condition that the elbow is extended in the pointing gesture toward the controlled device at an angle no less than a predefined angular threshold, and on condition that the hand is positioned within the interaction region and moved within the interaction region toward the controlled device and the gaze direction.

14. The product according to claim 13, wherein the instructions cause the computer to actuate the controlled device only when the elbow has been extended in the gesture for at least a predefined minimum time period.

15. The product according to claim 13, wherein the instructions cause the computer to receive a vocal command from the user, and to actuate the controlled device in response to the gesture and the vocal command.

16. The product according to claim 13, wherein the instructions cause the computer to detect, in the maps, a gaze direction of the user, and to actuate the controlled device in response to the gesture and the gaze direction.

17. The product according to claim 13, wherein the predefined angular threshold is at least 90°

, such that the controlled device is actuated only when the elbow is extended in the pointing gesture toward the controlled device at an angle greater than or equal to 90°

.