SYSTEMS AND METHODS FOR BIOMECHANICALLY-BASED EYE SIGNALS FOR INTERACTING WITH REAL AND VIRTUAL OBJECTS

• US 20150338915A1
• Filed: 05/09/2015
• Published: 11/26/2015
• Est. Priority Date: 05/09/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Eyefluence, Inc. (Alphabet Inc.)
• Inventors: Publicover, Nelson George, Marggraff, Lewis James, Drake, Eliot Francis
• Granted Patent: US 10,620,700 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 21/316 by observing the pattern of computer usage, e.g. typical user behaviour
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification
• CPC Class Codes: H04L 63/0861 : using biometrical features, e.g. fingerprint, retina-scan cryptographic mechanisms or cryptographic arrangements for entity authentication using biological data H04L9/3231
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/44504 : Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits
• CPC Class Codes: H04W 12/065 : Continuous authentication
• CPC Class Codes: H04W 12/33 : using wearable devices, e.g. using a smartwatch or smart-glasses

Abstract

Systems and methods are provided for discerning the intent of a device wearer primarily based on movements of the eyes. The system can be included within unobtrusive headwear that performs eye tracking and controls screen display. The system can also utilize remote eye tracking camera(s), remote displays and/or other ancillary inputs. Screen layout is optimized to facilitate the formation and reliable detection of rapid eye signals. The detection of eye signals is based on tracking physiological movements of the eye that are under voluntary control by the device wearer. The detection of eye signals results in actions that are compatible with wearable computing and a wide range of display devices.

Claims

1. A method for providing a graphical user interface to determine intent of a user based at least in part on movement of the user'"'"'s eye or eyes relative to an electronic display using a detector, comprising:

identifying when the gaze of at least one eye is directed at an object on the display;

identifying a saccadic movement of the at least one eye from the object towards a target location including a first icon on the display corresponding to an action;

confirming that the gaze of the at least one eye fixates at the target location at completion of the saccadic movement; and

performing the action on the object.

2. The method of claim 1, wherein the first icon is visually altered on the display prior to, during, or after the at least one eye fixates at the target location.

3. The method of claim 1, wherein the first icon is replaced with a second icon corresponding to the object at the target location.

4. The method of claim 3, wherein the second icon comprises a thumbnail of the object.

5. The method of claim 3, wherein the second icon comprises a likeness or a portion of the object.

6. The method of claim 1, further comprising providing visual confirmation on the display that the gaze of the at least one eye is directed at the object.

7. The method of claim 6, wherein the visual confirmation comprises presenting a border at least partially around the object.

8. The method of claim 6, wherein the visual confirmation comprises presenting a border at least partially around the object without visually modifying the object.

9. The method of claim 1, wherein identifying a saccadic movement comprises monitoring movement of the at least one eye to determine when an eye movement has a velocity above a predetermined threshold, thereby identifying that the eye movement is saccadic.

10. The method of claim 9, wherein the eye movement has a duration or distance above a predetermined threshold.

11. The method of claim 9, wherein identifying a saccadic movement further comprises determining a direction of the eye movement and, based on the velocity and direction of eye movement, predicting that the destination for the eye movement is the first target location.

12. The method of claim 1, further comprising presenting a plurality of images on the display, wherein the object is a first image of the images, wherein the second icon is a thumbnail of the first image, and wherein the action comprises replacing the plurality of images on the display with an enlarged version of the first image.

13. The method of claim 12, further comprising monitoring movement of the at least one eye relative to the enlarged version of the first image, identifying that the gaze of the at least one eye is directed towards a first region of the enlarged version, and moving the enlarged version of the first image to center the first region on the display.

14. The method of claim 1, wherein one or more of the steps are performed by a processing unit.

15. A system for providing a graphical user interface to determine intent of a user based at least in part on movement of the user'"'"'s eye or eyes, comprising:

a detector configured to monitor movement of at least eye of the user;

an electronic display operatively associated with the detector;

a processing unit operatively coupled to the detector and electronic display to;

identify when the gaze of the at least one eye is directed at an object on the display;

identify a saccadic movement of the at least one eye from the object towards target location including a first icon on the display corresponding to an action;

confirm that the gaze of the at least one eye fixates at the target location at completion of the saccadic movement; and

perform the action on the object.

16. The system of claim 15, further comprising a mobile electronic device, wherein the display is provided on the mobile electronic device, and wherein the detector comprises a camera on the display oriented such that user viewing the display is within a field of view of the camera.

17-35. -35. (canceled)

36. A system for providing a graphical user interface to determine intent of a user based on movement of the user'"'"'s eye or eyes, comprising:

a detector configured to monitor movement of at least one eye of a user;

an electronic display operatively associated with the detector;

a processing unit operatively coupled to the detector and electronic display to;

identify a first saccadic movement of the at least one eye towards a first target location including a first icon on the display corresponding;

replace the first icon on the display with a plurality of second icons at a plurality of second locations different than the first location;

confirm that the gaze of the at least one eye fixates at the first target location after the first saccadic movement; and

thereafter monitor the at least one eye to identify whether the at least one eye performs a second saccadic movement towards one of the plurality of second target locations.

37. The system of claim 36, wherein the processing unit is further configured to:

identify a second saccadic movement of the at least one eye from the first target location to one of the plurality of second locations;

confirm that the gaze of the at least one eye fixates at one of the second target locations at completion of the second saccadic movement to identify a selected icon; and

perform an action associated with the selected icon.

38. The system of claim 36, wherein the processing unit is configured to remove the first icon from the first target location before the gaze of the at least one eye fixates at the first target location.

39. The system of claim 36, wherein the plurality of second icons remain substantially stationary at the plurality of second locations.

40-61. -61. (canceled)