Gaze detection in a 3D mapping environment

• US 10,031,578 B2
• Filed: 09/05/2016
• Issued: 07/24/2018
• Est. Priority Date: 02/09/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bychkov, Eyal, Brezner, Oren, Galor, Micha, Or, Ofir, Pokrass, Jonathan, Hoffnung, Amir, Berliner, Tamir
• Primary Examiner(s): Sarma, Abhishek
• Application Number:
  US15/256,654
• Publication Number:
  US 20160370860A1
• Time in Patent Office: 687 Days
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A method includes receiving a sequence of three-dimensional (3D) maps of at least a part of a body of a user of a computerized system and extracting, from the 3D map, 3D coordinates of a head of the user. Based on the 3D coordinates of the head, a direction of a gaze performed by the user and an interactive item presented in the direction of the gaze on a display coupled to the computerized system are identified. An indication is extracted from the 3D maps an indication that the user is moving a limb of the body in a specific direction, and the identified interactive item is repositioned on the display responsively to the indication.

Claims

1. An apparatus, comprising:

a sensing device configured to receive a sequence of three dimensional (3D) maps of at least a part of a body of a user, including a head of the user, and to receive a two dimensional (2D) image of the user, the 2D image including an eye of the user; and

a computer coupled to the sensing device and configured to extract, from the 3D maps, 3D coordinates of the head of the user and to identify, based on the 3D coordinates of the head and the image of the eye, a direction of a gaze performed by the user, to identify an interactive item presented in the direction of the gaze on a display coupled to the computer, to extract from the 3D maps an indication that the user is moving a limb of the body in a specific direction, and to move the identified interactive item on the display responsively to the indication.

2. The apparatus according to claim 1, wherein the computer is configured to change a state of the interactive item.

3. The apparatus according to claim 2, wherein the computer is configured to change the state of the interactive item is changed responsively to the gaze.

4. The apparatus according to claim 2, wherein the computer is configured to change the state of the interactive item responsively to a vocal command received from the user.

5. The apparatus according to claim 2, wherein the computer is configured to change the state by directing input received from the user to the interactive item.

12. The method according to claim 1, wherein extracting the indication comprises detecting a swiping motion of a hand, and wherein repositioning the identified interactive item comprises moving the identified interactive item on the display in the direction of the swiping motion.

6. A method, comprising:

receiving a sequence of three-dimensional (3D) maps of at least a part of a body of a user of a computerized system;

extracting, from the 3D maps, 3D coordinates of a head of the user;

receiving a two dimensional (2D) image of the user, the image including an eye of the user;

identifying, based on the 3D coordinates of the head and the image of the eye, a direction of a gaze performed by the user;

identifying an interactive item presented in the direction of the gaze on a display coupled to the computerized system;

extracting from the 3D maps an indication that the user is moving a limb of the body in a specific direction; and

moving the identified interactive item on the display responsively to the indication.

7. The method according to claim 6, and comprising changing a state of the interactive item.

8. The method according to claim 7, wherein the state of the interactive item is changed responsively to the gaze.

9. The method according to claim 7, wherein the state of the interactive item is changed responsively to a vocal command received from the user.

10. The method according to claim 7, wherein changing the state comprises directing input received from the user to the interactive item.

11. The method according to claim 7, and comprising identifying a target point on the display in the direction of the gaze, and calculating a calibration coefficient based on the proximity of the target point to the interactive item.

13. The method according to claim 6, wherein extracting the indication comprises detecting a pointing motion of a finger, and wherein repositioning the identified interactive item comprises moving the identified interactive item on the display in the direction pointed to by the finger.

14. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a sequence of three-dimensional (3D) maps of at least a part of a body of a user of the computer, including a head of the user, and to receive a two dimensional (2D) image of the user, the 2D image including an eye of the user, to extract, from the 3D maps, 3D coordinates of the head of the user, to identify, based on the 3D coordinates of the head and the image of the eye, a direction of a gaze performed by the user, to identify an interactive item presented in the direction of the gaze on a display coupled to the computer, to extract from the 3D maps an indication that the user is moving a limb of the body in a specific direction, and to move the identified interactive item on the display responsively to the indication.