Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects

• US 9,823,744 B2
• Filed: 11/02/2015
• Issued: 11/21/2017
• Est. Priority Date: 05/09/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Publicover, Nelson George, Marggraff, Lewis James, Drake, Eliot Francis, Connaughton, Spencer James
• Primary Examiner(s): He, Weiming
• Application Number:
  US14/930,617
• Publication Number:
  US 20160085302A1
• Time in Patent Office: 750 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 21/316 by observing the pattern of computer usage, e.g. typical user behaviour
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification
• CPC Class Codes: H04L 63/0861 : using biometrical features, e.g. fingerprint, retina-scan cryptographic mechanisms or cryptographic arrangements for entity authentication using biological data H04L9/3231
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/44504 : Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits
• CPC Class Codes: H04W 12/065 : Continuous authentication
• CPC Class Codes: H04W 12/33 : using wearable devices, e.g. using a smartwatch or smart-glasses

Abstract

Systems and methods are provided for discerning the intent of a device wearer primarily based on movements of the eyes. The system can be included within unobtrusive headwear that performs eye tracking and controls screen display. The system can also utilize remote eye tracking camera(s), remote displays and/or other ancillary inputs. Screen layout is optimized to facilitate the formation and reliable detection of rapid eye signals. The detection of eye signals is based on tracking physiological movements of the eye that are under voluntary control by the device wearer. The detection of eye signals results in actions that are compatible with wearable computing and a wide range of display devices.

Claims

1. A method for providing a graphical user interface to determine intent of a user based at least in part on movement of an eye or eyes of the user, the method comprising:

displaying a plurality of menu items on a display;

identifying, with a detector, when at least one eye of the user is directed at a first menu item of the plurality of menu items;

identifying, with the detector, a saccade of the at least one eye of the user from the first menu item towards an activation icon on the display;

confirming, with the detector, that the saccade of the at least one eye of the user is completed to within a predetermined distance from a location of the activation icon; and

initiating performance of one or more actions in relation to the first menu item,

wherein a visual confirmation is presented on the display indicating that the at least one eye of the user has completed the saccade to within the predetermined distance from the location of the activation icon, wherein the visual confirmation comprises replacing the activation icon with an icon corresponding to the first menu item.

2. The method of claim 1, wherein a visual confirmation is presented on the display indicating that the at least one eye of the user is directed at the first menu item.

3. The method of claim 2, wherein the visual confirmation comprises one of the group consisting of presenting a border at least partially around the first menu item, changing brightness of the first menu item, changing color of the first menu item, adding an indicator in a region of the first menu item, and adding imagery on the display that augments the first menu item.

4. The method of claim 1, wherein the plurality of menu items comprise a plurality of images, wherein the first menu item is one of the plurality of images, and wherein the action comprises replacing the plurality of images on the display with an image corresponding to the first menu item.

5. The method of claim 1, wherein the icon comprises one of the group consisting of a thumbnail of the first menu item, a representation of the first menu item, a likeness of the first menu item, an image of the first menu item, and a portion of the first menu item.

6. The method of claim 1, wherein the one or more actions comprise identifying the first menu item from a database of object templates.

7. The method of claim 1, wherein the one or more actions comprise purchasing an item corresponding to the first menu item.

8. The method of claim 1, wherein the one or more actions comprise retrieving first menu item information related to the identified first menu item from one or more databases of information about identified objects.

9. The method of claim 8, wherein the one or more actions further comprise displaying the first menu item information on the display.

10. The method of claim 8, wherein the one or more actions action further comprise performing one or more additional actions related to the first menu item information.

11. The method of claim 10, wherein the one or more additional actions comprise sending information regarding an identity of the user to a processor that is operatively coupled to the identified first menu item.

12. The method of claim 1, wherein the performance of the one or more actions is initiated within less than 250 milliseconds after the saccade is completed within the predetermined distance from the location of the activation icon.

13. The method of claim 1, wherein the performance of the one or more actions is initiated within less than 200 milliseconds after the saccade is completed within the predetermined distance from the location of the activation icon.

14. The method of claim 1, wherein the one or more actions comprise launching an application associated with the first menu item.

15. The method of claim 1, wherein the plurality of menu items comprise icons associated with respective applications.

16. The method of claim 1, wherein the activation icon is presented at a fixed location on the display independent from a field on the display including the plurality of menu items.

17. The method of claim 1, wherein each of the plurality of menu items is associated with a different action and wherein the one or more actions comprises initiating performance of an action associated with the first menu item that is different than actions performed by other menu items of the plurality of menu items.

18. A system for providing a graphical user interface to determine intent of a user, the system comprising:

a detector configured to monitor movement of at least one eye of the user;

an electronic display operatively associated with the detector;

a processing unit operatively coupled to the detector and electronic display; and

a module stored in memory, the module to configure the processing unit to;

display a plurality of menu items on the display;

identify when at least one eye of the user is directed at a first menu item of the plurality of menu items on the display;

identify a saccade of the at least one eye of the user from the first menu item towards an activation icon on the display;

confirm that the saccade of the at least one eye is completed to within a predetermined distance from a location of the activation icon on the display; and

initiate performance of one or more actions in relation to the first menu item including presenting a visual confirmation on the display indicating that the at least one eye of the user has completed the saccade to within the predetermined distance from the location of the activation icon, wherein the visual confirmation comprises replacing the activation icon with an icon corresponding to the first menu item.

19. The system of claim 18, wherein the module further is to configure the processing unit to provide visual confirmation on the display that the at least one eye of the user is directed at the first menu item.

20. The system of claim 18, wherein the module is to configure the processing unit to initiate performance of the one or more actions within less than 250 milliseconds after the saccade is completed within the predetermined distance from the location of the activation icon.

21. A system for providing a graphical user interface to determine intent of a user, the system comprising:

a detector configured to monitor movement of at least one eye of the user;

a processing unit operatively coupled to the detector and a display; and

a module stored in memory, the module to configure the processing unit to;

display a plurality of menu items on the display;

identify when at least one eye of the user is directed at a first menu item of the plurality of menu items at a first location on the display;

identify a saccade of the at least one eye of the user from the first menu item towards an activation icon at a second location on the display;

confirm that the saccade of the at least one eye is completed to within a predetermined distance from the second location of the activation icon; and

initiate performance of one or more actions in relation to the first menu item, the one or more actions comprising replacing the activation icon at the second location with an icon corresponding to the first menu item.

22. The system of claim 21, wherein initiating performance of the one or more actions further comprises initiating an action associated with the first menu item.

23. The system of claim 21, wherein the icon comprises one of the group consisting of a thumbnail of the first menu item, a representation of the first menu item, a likeness of the first menu item, an image of the first menu item, and a portion of the first menu item.

24. A method for providing a graphical user interface to determine intent of a user, the method comprising:

displaying a plurality of menu items on a display;

identifying, with a detector, when at least one eye of the user is directed at a first menu item of the plurality of menu items at a first location;

identifying, with the detector, a saccade of the at least one eye of the user from the first location towards an activation icon at a second location on the display;

confirming, with the detector, that the saccade of the at least one eye of the user is completed from the first menu item to within a predetermined distance from the second location of the activation icon; and

initiating performance of one or more actions in relation to the first menu item, the one or more actions comprising replacing the activation icon at the second location with an icon corresponding to the first menu item.

25. The method of claim 24,

wherein the menu items comprise a plurality of images, wherein the first menu item is a first image of the plurality of images;

and

wherein the one or more actions further comprise presenting an image of the first image on the display.

26. The method of claim 24, wherein initiating performance of the one or more actions further comprises initiating an action associated with the first menu item.

27. The method of claim 24, wherein the icon comprises one of the group consisting of a thumbnail of the first menu item, a representation of the first menu item, a likeness of the first menu item, an image of the first menu item, and a portion of the first menu item.

28. The method of claim 24, wherein the one or more actions comprise launching an application associated with the first menu item.