Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects

• US 10,564,714 B2
• Filed: 08/15/2016
• Issued: 02/18/2020
• Est. Priority Date: 05/09/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Marggraff, Lewis James, Publicover, Nelson George, Connaughton, Spencer James, Lord, Nathan, Milford, Peter, Maleev, Ivan
• Primary Examiner(s): Naghdali, Khalil
• Application Number:
  US15/237,581
• Publication Number:
  US 20170123492A1
• Time in Patent Office: 1,282 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04L 63/0861 using biometrical features, e.g. fingerprint, retina-scan cryptographic mechanisms or cryptographic arrangements for entity authentication using biological data H04L9/3231
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 12/33 : using wearable devices, e.g. using a smartwatch or smart-glasses

Abstract

Systems and methods are provided for discerning the intent of a device wearer primarily based on movements of the eyes. The system may be included within unobtrusive headwear that performs eye tracking and controls screen display. The system may also utilize remote eye tracking camera(s), remote displays and/or other ancillary inputs. Screen layout is optimized to facilitate the formation and reliable detection of rapid eye signals. The detection of eye signals is based on tracking physiological movements of the eye that are under voluntary control by the device wearer. The detection of eye signals results in actions that are compatible with wearable computing and a wide range of display devices.

Claims

1. A method for providing a graphical user interface to convey intent of a user based at least in part on movement of the user'"'"'s one or both eyes using a detector, the method comprising:

identifying, with the detector, when the user'"'"'s one or both eyes are directed at a first key of a first keyboard;

identifying, with the detector, a first one or more saccades of the user'"'"'s one or both eyes from the first key towards, not within a pre-determined distance from, a second key of a second keyboard that has a set of keys;

confirming, with the detector, that one or more corrective saccades toward the location of the second key for the first one or more saccades are completed to within the pre-determined distance from the location of the second key; and

performing an action associated with one or both of the first key and the second key, without waiting for perception of the second key by the user.

2. The method of claim 1, wherein the second keyboard is one of a virtual keyboard projected onto a display and a physical object.

3. The method of claim 1, wherein the action comprises concatenating a character associated with the first key to a series of characters.

4. The method of claim 3, wherein the concatenating a character associated with the first key forms one of a group consisting of a word, phrase, sentence, and a number.

5. The method of claim 1, wherein the first and second keyboards are virtual keyboards presented on a display adjacent one another.

6. The method of claim 1, wherein the first and second keyboards include a plurality of keys including one or more of a representation of numbers, letters, words, phrases, selections, media glyphs, or special characters.

7. The method of claim 1, wherein the action comprises adding a symbol assigned to the first key to a field displayed on a display.

8. The method of claim 1, further comprising:

identifying the second key with the detector when the user'"'"'s one or both eyes are directed at the second keyboard;

identifying, with the detector, a second one or more saccades of the user'"'"'s one or both eyes from the second key towards a third key of the first keyboard;

confirming, with the detector, that a second one or more corrective saccades toward a location of the third key for the second one or more saccades are completed within a predetermined distance from the third key of the first keyboard; and

performing a second action associated with one or both the third key and the second key, without waiting for perception of the third key of the first keyboard by the user.

9. The method of claim 8, wherein the action comprises modifying symbols presented on keys of the second keyboard based on at least in part on the first key.

10. The method of claim 9, wherein modifying symbols presented on keys of the second keyboard comprises omitting symbols from the keys of the second keyboard that cannot be associated with a symbol presented on the first key.

11. The method of claim 8, wherein the action comprises modifying symbols presented on keys of the second key board based on at least in part on the first key.

12. The method of claim 1, wherein the second key is an activation target.

13. The method of claim 1, wherein the second key is located within one of a parafoveal view and a peripheral view of the user.

14. The method of claim 1, wherein the first key is located within a virtual keyboard on a display and the second key is located within a field-of-view of the user not on the display.

15. The method of claim 1, wherein the action is performed immediately upon confirming that the one or more corrective saccades for the first one or more saccades are completed from the first key to within a pre-determined distance from the location of the second key independent of any further action by the user.

16. The method of claim 1, further comprising:

after identifying when the user'"'"'s one or both eyes are directed at the first key of the first keyboard, changing a brightness of a most frequently used feature of the second key of the second keyboard.

17. The method of claim 1, wherein:

the second keyboard has a same set of keys in an identical layout as the first keyboard; and

the second keyboard is adjacent and in a same orientation as the first keyboard.

18. A method for providing a graphical user interface to convey intent of a user based at least in part on movement of the user'"'"'s one or both eyes using a first keyboard, a second keyboard, and a detector, the method comprising:

identifying, with the detector, when the user'"'"'s one or both eyes are directed within a predetermined distance from a location of a first key on the first or second keyboard, wherein at least one key of the first or second keyboard is activatable using a single saccadic eye movement while other keys of the first and second keyboards are activatable using two saccadic eye movements, wherein the second keyboard is positioned vertically adjacent and in a same orientation as the first keyboard relative to the user'"'"'s eyes, the second keyboard having a same set of keys in a same layout as the first keyboard, and wherein most frequently used keys of one or more of the first and second keyboards occupy one or more central rows of a plurality of rows of keys of the keyboards;

identifying, with the detector, a first one or more saccades of the user'"'"'s one or both eyes from the first key towards, not within a pre-determined distance from, a second key on the keyboard;

confirming, with the detector, that one or more corrective saccades toward a location of the second key for the one or more saccades are completed to within the pre-determined distance from the location of the second key; and

performing an action associated with one or both of the first key and the second key.

19. The method of claim 18, wherein the action is performed without waiting for perception of the second key by the user.

20. The method of claim 18, wherein the action comprises concatenating a character associated with the first key to a stored series of characters.

21. The method of claim 20, wherein the concatenating a character associated with the first key forms one of a group consisting of a word, a sentence, and a number.

22. The method of claim 18, wherein at least one of the first and second keyboards are projected onto the graphical user interface.

23. The method of claim 18, wherein the first and second keyboards include an off-display modifier target for a frequently used function associated with at least one key of one of the keyboards, the method further comprising:

changing brightness of a most frequently used feature of the second key on the keyboard after identifying with the detector when the user'"'"'s one or both eyes are within the predetermined distance from the location of the first key on the keyboard.

24. A graphical user interface to determine intent of a user based at least in part on movement of the user'"'"'s one or both eyes, the graphical user interface comprising:

a projector configured to display at least a first keyboard;

a detector configured to;

identify when the user'"'"'s one or both eyes are directed at a first key of the first keyboard, wherein at least one key of the first keyboard is activatable using a single saccadic eye movement while other keys of the first keyboard are activatable using two saccadic eye movements;

identify a first one or more saccades of the user'"'"'s one or both eyes from the first key towards, not within a pre-determined distance from, a second key of a second keyboard;

confirm that one or more corrective saccades toward a location of the second key for the first one or more saccades are completed to within a pre-determined distance from the location of the second key; and

a processor configured to perform an action associated with one or both of the first key and the second key in response to confirmation that the one or more corrective saccades for the first one or more saccades are completed to within the pre-determined distance from the location of the second key, without waiting for perception of the second key by the user.

25. The graphical user interface of claim 24, wherein the action the processor is configured to perform comprises concatenating a character associated with the first key to a series of characters.

26. The graphical user interface of claim 24, wherein the first and second keyboards include a plurality of keys including one or more of a representation of numbers, letters, words, phrases, selections, media glyphs, or special characters.

27. The graphical user interface of claim 24, wherein the first and second keyboards are virtual keyboards that have identical layouts.

28. The graphical user interface of claim 24, wherein the detector is configured to:

identify the second key when the user'"'"'s one or both eyes are directed at the second keyboard;

identify a second one or more saccades of the user'"'"'s one or both eyes from the second key towards a third key of the first keyboard;

confirm that a second one or more corrective saccades for the second one or more saccades are completed within a specified distance from the third key of the first keyboard; and

the processor is configured to perform a second action associated with one or both the third key and the second key in response to confirmation that the second one or more corrective saccades for the second one or more saccades are completed within the specified distance from the third key of the first keyboard, without waiting for perception of the third key of the first keyboard by the user.