SYSTEMS AND METHODS FOR BIOMECHANICALLY-BASED EYE SIGNALS FOR INTERACTING WITH REAL AND VIRTUAL OBJECTS

• US 20170123492A1
• Filed: 08/15/2016
• Published: 05/04/2017
• Est. Priority Date: 05/09/2014
• Status: Active Grant

• Current Assignee: Eyefluence, Inc. (Alphabet Inc.)
• Original Assignee: Eyefluence, Inc. (Alphabet Inc.)
• Inventors: Marggraff, Lewis James, Publicover, Nelson George, Connaughton, Spencer James, Milford, Peter, Maleev, Ivan, Lord, Nathan
• Granted Patent: US 10,564,714 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04L 63/0861 using biometrical features, e.g. fingerprint, retina-scan cryptographic mechanisms or cryptographic arrangements for entity authentication using biological data H04L9/3231
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 12/33 : using wearable devices, e.g. using a smartwatch or smart-glasses

Abstract

Systems and methods are provided for discerning the intent of a device wearer primarily based on movements of the eyes. The system may be included within unobtrusive headwear that performs eye tracking and controls screen display. The system may also utilize remote eye tracking camera(s), remote displays and/or other ancillary inputs. Screen layout is optimized to facilitate the formation and reliable detection of rapid eye signals. The detection of eye signals is based on tracking physiological movements of the eye that are under voluntary control by the device wearer. The detection of eye signals results in actions that are compatible with wearable computing and a wide range of display devices.

Claims

1-9. -9. (canceled)

10. A method for providing a graphical user interface to convey intent of a user based at least in part on movement of the user'"'"'s one or both eyes using a detector, the method comprising:

identifying, with the detector, when the user'"'"'s one or both eyes are directed at a first key of a first object;

identifying, with the detector, a first one or more saccades of the user'"'"'s one or both eyes from the first key towards a second key of a second object;

confirming, with the detector, that the first one or more saccades are completed to within a pre-determined distance from a location of the second key; and

performing an action associated with one or both of the first key and the second key, without waiting for perception of the second key by the user.

11. The method of claim 10, wherein the first object is one of a virtual object projected onto a display and a physical object.

12. The method of claim 10, wherein the second object is one of a virtual keyboard projected onto a display and a physical object.

13. The method of claim 10, wherein the action comprises concatenating a character associated with the first key to a series of characters.

14. The method of claim 13, wherein the concatenating a character associated with the first key forms one of a group consisting of a word, phrase, sentence, and a number.

15. The method of claim 10, wherein the first and second objects are virtual keyboards presented on a display adjacent one another.

16. The method of claim 10, wherein the first and second objects include a plurality of keys including one or more of a representation of numbers, letters, words, phrase, selections, media glyphs, or special characters.

17. The method of claim 10, wherein the first and second objects are keyboards that have identical layouts.

18. The method of claim 10, wherein the first and second objects are keyboards that have different layouts than one another.

19. The method of claim 10, wherein the action comprises adding an symbol assigned to the first key to a field displayed on a display.

21. The method of claim 10, further comprising:

identifying the second key with the detector when the user'"'"'s one or both eyes are directed at the second object;

identifying, with the detector, a second one or more saccades of the user'"'"'s one or both eyes from the second key towards a third key of the first object;

confirming, with the detector, that the second one or more saccades are completed within a predetermined distance from the third key of the first object; and

performing a second action associated with one or both the third key and the second key, without waiting for perception of the third of the first object by the user.

22. The method of claim 21, wherein the action comprises modifying symbols presented on keys of the second object based on at least in part on the first key.

23. The method of claim 22, wherein modifying symbols presented on keys of the second object comprises omitting symbols from the keys of the second object that cannot be associated with a symbol presented on the first key.

179. The method of claim 21, wherein the action comprises modifying symbols presented on keys of the second object based on at least in part on the first key.

20. (canceled)

24. (canceled)

25. (canceled)

26. (canceled)

27. (canceled)

28. (canceled)

29. A method for providing a graphical user interface to convey intent of a user based at least in part on movement of the user'"'"'s one or both eyes using a keyboard and a detector, the method comprising:

identifying, with the detector, when the user'"'"'s one or both eyes are directed at a first key on the keyboard;

identifying, with the detector, a first one or more saccades of the user'"'"'s one or both eyes from the first key towards a second key on the keyboard;

confirming, with the detector, that the first one or more saccades is completed to within a pre-determined distance from a location of the second key; and

performing an action associated with one or both of the first key and the second key without waiting for perception of the second key by the user.

30. The method of claim 29, wherein the keyboard is one of a virtual keyboard projected onto a display and a physical keyboard.

31. The method of claim 29, wherein the second key is an activation target.

32. The method of claim 29, wherein the action comprises concatenating an character associated with the first key to a series of characters.

33. The method of claim 32, wherein the concatenating an character associated with the first key forms one of group consisting of a word, a phrase, a sentence, and a number.

35. The method of claim 29, wherein the second key is located within one of a parafoveal view and a peripheral view of the user.

36. The method of claim 29, wherein the first key is located within a virtual keyboard on a display and the second key is located within a field-of-view of the user not on the display.

37. The method of claim 29, wherein the action is performed immediately upon confirming that the first one or more saccades are completed from the first key to within a pre-determined distance from the location of the second key independent of any further action by the user.

34. (canceled)

38. (canceled)

39. (canceled)

40. A method for providing a graphical user interface to convey intent of a user based at least in part on movement of the user'"'"'s one or both eyes using a keyboard and a detector, the method comprising:

identifying, with the detector, when the user'"'"'s one or both eyes are directed within a predetermined distance from a location of a first key on the keyboard;

identifying, with the detector, a first one or more saccades of the user'"'"'s one or both eyes from the first key towards a second key on the keyboard;

confirming, with the detector, that the one or more saccades are completed to within a pre-determined distance from a location of the second key; and

performing an action associated with one or both of the first key and the second key.

41. The method of claim 40, wherein the action is performed without waiting for perception of the second key by the user.

42. The method of claim 40, wherein the action comprises concatenating a character associated with the first key to a stored series of characters.

43. The method of claim 42, wherein the concatenating a character associated with the first key forms one of a group consisting of a word, a sentence, and a number.

44. The method of claim 40, wherein the keyboard is projected onto the graphical user interface.

45-171. -171. (canceled)

172. A graphical user interface to determine intent of a user based at least in part on movement of the user'"'"'s one or both eyes, the graphical user interface comprising:

a projector configured to display at least a first object;

a detector configured to;

identify when the user'"'"'s one or both eyes are directed at a first key of the first object;

identify a first one or more saccades of the user'"'"'s one or both eyes from the first key towards a second key of a second object;

confirm that the first one or more saccades are completed to within a pre-determined distance from a location of the second key; and

a processor configured to perform an action associated with one or both of the first key and the second key in response to confirmation that the first one or more saccades are completed to within the pre-determined distance from a location of the second key, without waiting for perception of the second key by the user.

173. The graphical user interface of claim 172, wherein the projector is configured to project onto a display a virtual keyboard as the first object.

174. The graphical user interface of claim 172, wherein the second object is a virtual keyboard projected by the projector onto a display or a physical object.

175. The graphical user interface of claim 172, wherein the action the processor is configured to perform comprises concatenating a character associated with the first key to a series of characters.

176. The graphical user interface of claim 172, wherein the first and second objects include a plurality of keys including one or more of a representation of numbers, letters, words, phrases, selections, media glyphs, or special characters.

177. The graphical user interface of claim 172, wherein projector configured to display the first and second objects, the first and second objects being virtual keyboards that have identical layouts.

178. The graphical user interface of claim 172, wherein the detector is configured to:

identify the second key when the user'"'"'s one or both eyes are directed at the second object;

identify a second one or more saccades of the user'"'"'s one or both eyes from the second key towards a third key of the first object;

confirm that the second one or more saccades are completed within a specified distance from the third key of the first object; and

the processor is configured to perform a second action associated with one or both the third key and the second key in response to confirmation that the second one or more saccades are completed within the specified distance from the third key of the first object, without waiting for perception of the third key of the first object by the user.