SYSTEMS AND METHODS FOR BIOMECHANICALLY-BASED EYE SIGNALS FOR INTERACTING WITH REAL AND VIRTUAL OBJECTS

• US 20160274660A1
• Filed: 04/18/2016
• Published: 09/22/2016
• Est. Priority Date: 05/09/2014
• Status: Active Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Eyefluence, Inc. (Alphabet Inc.)
• Inventors: Publicover, Nelson George, Marggraff, Lewis James, Drake, Eliot Francis, Connaughton, Spencer James
• Application Number:
  US15/131,273
• Publication Number:
  US 20160274660A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 21/316 by observing the pattern of computer usage, e.g. typical user behaviour
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification
• CPC Class Codes: H04L 63/0861 : using biometrical features, e.g. fingerprint, retina-scan cryptographic mechanisms or cryptographic arrangements for entity authentication using biological data H04L9/3231
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/44504 : Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits
• CPC Class Codes: H04W 12/065 : Continuous authentication
• CPC Class Codes: H04W 12/33 : using wearable devices, e.g. using a smartwatch or smart-glasses

Abstract

Systems and methods are provided for discerning the intent of a device wearer primarily based on movements of the eyes. The system can be included within unobtrusive headwear that performs eye tracking and controls screen display. The system can also utilize remote eye tracking camera(s), remote displays and/or other ancillary inputs. Screen layout is optimized to facilitate the formation and reliable detection of rapid eye signals. The detection of eye signals is based on tracking physiological movements of the eye that are under voluntary control by the device wearer. The detection of eye signals results in actions that are compatible with wearable computing and a wide range of display devices.

Claims

1-30. -30. (canceled)

31. A method for providing a graphical user interface to determine intent of a user based at least in part on movement of the user'"'"'s one or both eyes using a detector, comprising:

identifying, with the detector, when the user'"'"'s one or both eyes are directed at an object at an object location;

identifying, with the detector, a first saccade of the user'"'"'s one or both eyes from the object location towards a target at a target location;

identifying, with the detector, one or more corrective saccades of the user'"'"'s one or both eyes moving closer towards the target location;

confirming, with the detector, that at least one of the corrective saccades is completed within a pre-determined distance from the target location; and

performing an action related to one or more of the object, the object location, the target, and the target location.

32. The method of claim 31, wherein identifying, with the detector, one or more corrective saccades of the user'"'"'s one or both eyes moving closer towards the target location comprises one or more of:

identifying a first length of the first saccade and determining that a second saccade has a smaller length than the first saccade;

determining when the user'"'"'s one or both eyes moving closer towards the target location has an angular velocity greater than a pre-determined lower velocity threshold and less than a predetermined upper velocity threshold; and

identifying a second length of a second saccade and determining when the second length is greater than a pre-determined lower length threshold and less than a predetermined upper length threshold.

33. The method of claim 31, wherein the first saccade is a memory-guided saccade.

34. A method for providing a graphical user interface to determine intent of a user based at least in part on movement of the user'"'"'s one or both eyes using a detector, comprising:

identifying, with the detector, when the user'"'"'s one or both eyes are directed at a first object at a first object location;

identifying, with the detector, a saccade of the user'"'"'s one or both eyes from the first object location towards a target object at a target location on a display;

confirming, with the detector, that one of a predicted landing location and an actual landing location of the saccade is within a pre-determined distance from the target location; and

performing an action that includes replacing the target object with an icon at the target location related to one of the first object, and the first object location.

35. The method of claim 34, wherein one or more additional actions are performed upon confirming that one of a predicted landing location and a measured landing location of the saccade is within a pre-determined distance from the target location.

36. The method of claim 34, wherein the saccade is a memory-guided saccade.

37. The method of claim 34, wherein the icon comprises one of the first object, a thumbnail of the first object, a likeness of the first object, a portion of text in a region of the first object location, an image of the first object location, and a portion of an image of the first object location.

38. The method of claim 34, wherein the icon comprises one of the first object, and the first object location familiar to the user, thereby reducing a cognitive load of the user.

39. The method of claim 38, whereby reducing the user'"'"'s cognitive load reduces the time required to produce an additional intent by the user.

40. The method of claim 34, wherein replacing the target object with an icon is performed prior to completion of the saccade.

41. The method of claim 34, wherein replacing the target object with an icon is performed prior to completion of the saccade to produce a gap effect, thereby reducing time required to produce an additional intent by the user.

42. The method of claim 34, wherein the action is performed without waiting for perception of the target object.

43. A method for providing a graphical user interface to determine intent of a user based at least in part on movement of the user'"'"'s head and one or both of the user'"'"'s eyes using a head movement detector and an eye movement detector, comprising:

identifying, with the head movement detector, when the user'"'"'s head moves at a head velocity;

identifying at substantially the same time, with the eye movement detector, when the one or both of the user'"'"'s eyes move at an eye velocity;

identifying, based at least in part on the head velocity and the eye velocity, a vestibulo-ocular movement of the one or both of the user'"'"'s eyes;

confirming that the one or both of the user'"'"'s eyes are directed at a viewed object at a viewed object location based at least in part on the determined vestibulo-ocular movement of the one or both of the user'"'"'s eyes; and

performing an action related to one or more of the viewed object, and the viewed object location.

44. The method of claim 43, wherein identifying the vestibulo-ocular eye movement comprises:

calculating, based on the head velocity, a head direction and a head magnitude;

calculating, based on the eye velocity, an eye direction and an eye magnitude; and

confirming that the head direction is opposite the eye direction within a predetermined direction threshold, and the head magnitude and the eye magnitude are similar within a predetermined magnitude threshold.

45. The method of claim 44, wherein the action is further related to one or more of the eye direction, the head direction, the eye magnitude, and the head magnitude.

46. The method of claim 43, wherein the head movement detector is one of a scene camera affixed to a headwear worn by the user and directed at the environment of the user, and an accelerometer affixed to the headwear.

47. The method of claim 43, wherein the eye movement detector comprises one or more cameras affixed to a headwear worn by the user and directed toward the one or both of the user'"'"'s eyes.

48. The method of claim 43, wherein the viewed object is one of a virtual object shown on a display, and a real object.

49. The method of claim 48, wherein the display is one of an augmented reality display, a virtual reality display, a mobile display device, a sign, and a display monitor.

50. The method of claim 49, wherein the action includes showing an indicator on the display within a predetermined viewing distance from the viewed object location.

51. The method of claim 50, wherein the indicator is one of one or more alphanumeric characters, halos, cursors, arrows, and symbols.

52. A method for providing a graphical user interface to determine intent of a user based at least in part on movements of the user'"'"'s head and one or both of the user'"'"'s eyes using a head movement detector and an eye movement detector, comprising:

identifying, with the eye movement detector, an initial viewed object at an initial viewed object location;

identifying, with the head movement detector, when the user'"'"'s head moves at a head velocity;

identifying at substantially the same time, with the eye movement detector, when the one or both of the user'"'"'s eyes move at an eye velocity;

identifying, based on the eye velocity, a first saccadic movement of the eye;

confirming, either upon completion of the head movement and the first saccadic eye movement or before completion of the head movement and the first saccadic eye movement, that the one or both of the user'"'"'s eyes are directed at a target object at a target object location; and

performing an action related to one or more of the initial viewed object, the initial viewed object location, the target object, and the target object location.

53. The method of claim 52, wherein the action is performed without waiting for perception of the target object by the user.

54. The method of claim 55, wherein the action is performed within 250 milliseconds after completion of the first saccadic movement.

55. The method of claim 52, wherein identifying the first saccade comprises determining when the eye velocity is greater than a predetermined velocity threshold.

56. The method of claim 52, wherein the initial viewed object is one of a real object, and a virtual object shown on a display.

57. The method of claim 52, wherein the target object is one of a real object, and a virtual object shown on a display.

58. The method of claim 52, wherein the target object is virtual object shown on a display and wherein the action includes removing the target object from the display.

59. The method of claim 52, wherein the action includes showing one or more additional target virtual objects on a display.

60. The method of claim 52, further comprising:

identifying, with the eye movement detector, one or more corrective saccadic eye movements following the first saccadic eye movement;

confirming, either upon completion of the one or more corrective saccadic eye movements or before completion of the one or more corrective saccadic eye movements, that the at least one eye of the user is directed at a corrected target object at a corrected target object location; and

performing an action related to one of the corrected target object, and the corrected target object location.