Gaze-based control of device operations

• US 10,346,985 B1
• Filed: 12/01/2017
• Issued: 07/09/2019
• Est. Priority Date: 10/15/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Poliakov, Fedir
• Primary Examiner(s): Liu, Li
• Application Number:
  US15/829,539
• Time in Patent Office: 585 Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification

Abstract

Systems, devices, media, and methods are presented for gaze-based control of device operations. One method includes receiving a video stream from an imaging device, the video stream depicting one or more eyes, determining a gaze direction for the one or more eyes depicted in the video stream, detecting a change in the gaze direction of the one or more eyes, and triggering an operation in a client device based on the change in the gaze direction.

Claims

1. A method comprising:

receiving a video stream from an imaging device, the video stream depicting one or more eyes;

determining a gaze direction for the one or more eyes depicted in the video stream relative to a display device of a client device by calculating a cone having a radius and angle based on an appearance of the one or more eyes;

detecting a change in the gaze direction of the one or more eyes; and

triggering an operation in the client device responsive to the change in the gaze direction exceeding a predetermined percentage threshold of a viewable area of the display device.

2. The method of claim 1, wherein the determining comprises identifying a location of the one or more eyes relative to the imaging device.

3. The method of claim 2, wherein the determining further comprises forming a triangle extending between the imaging device, the display device, and the one or more eyes.

4. The method of claim 2, wherein the determining further comprises calculating a line of sight extending from the one or more eyes to the display device.

5. The method of claim 4, further comprising identifying an object or area of interest on the display device based on the line of sight.

6. The method of claim 1, wherein the triggering comprises performing a predetermined operation associated with an object or area of interest in a region of a display device associated with the change in gaze direction.

7. A method comprising:

receiving a video stream from an imaging device, the video stream depicting one or more eyes;

determining a gaze direction for the one or more eyes depicted in the video stream relative to a display device of a client device;

detecting a change in the gaze direction of the one or more eyes followed by identifying an obstruction of at least one of the one or more eyes; and

triggering an operation in the client device responsive to the detected change in the gaze direction exceeding a predetermined percentage threshold of a viewable area of the display device followed by the identified obstruction of the at least one of the one or more eyes exceeding a predetermined length of time.

8. The method of claim 7, wherein the triggering comprises performing a predetermined operation associated with the detection of the change in the gaze direction followed by the identification of the obstruction.

9. A non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving a video stream from an imaging device, the video stream depicting one or more eyes;

determining a gaze direction for the one or more eyes depicted in the video stream relative to a display device of a client device by calculating a cone having a radius and angle based on an appearance of the one or more eyes;

detecting a change in the gaze direction of the one or more eyes; and

triggering an operation in the client device responsive to the change in the gaze direction exceeding a predetermined percentage threshold of a viewable area of the display device.

10. The non-transitory machine-readable storage medium of claim 9, wherein the determining comprises identifying a location of the one or more eyes relative to the imaging device.

11. The non-transitory machine-readable storage medium of claim 10, wherein the determining further comprises forming a triangle extending between the imaging device, the display device, and the one or more eyes.

12. The non-transitory machine-readable storage medium of claim 10, wherein the determining further comprises calculating a line of sight extending from the one or more eyes to the display device.

13. The non-transitory machine-readable storage medium of claim 12, further comprising identifying an object or area of interest on the display device based on the line of sight.

14. The non-transitory machine-readable storage medium of claim 9, wherein the triggering comprises performing a predetermined operation associated with an object or area of interest in a region of a display device associated with the change in gaze direction.

15. A non-transitory machine-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving a video stream from an imaging device, the video stream depicting one or more eyes;

determining a gaze direction for the one or more eyes depicted in the video stream relative to a display device of a client device;

detecting a change in the gaze direction of the one or more eyes followed by identifying an obstruction of at least one of the one or more eyes; and

triggering an operation in the client device responsive to the detected change in the gaze direction exceeding a predetermined percentage threshold of a viewable area of the display device followed by the identified obstruction of the at least one of the one or more eyes exceeding a predetermined length of time.

16. The non-transitory machine-readable storage medium of claim 15, wherein the triggering comprises performing a predetermined operation associated with the detection of the change in the gaze direction followed by the identification of the obstruction.