Human-computer interface using high-speed and accurate tracking of user interactions

• US 10,664,050 B2
• Filed: 09/21/2018
• Issued: 05/26/2020
• Est. Priority Date: 09/21/2018
• Status: Active Grant

• Current Assignee: Neurable Inc.
• Original Assignee: Neurable Inc.
• Inventors: Alcaide, Ramses, Padden, Dereck, Jantz, Jay, Hamet, James, Morris, Jr., Jeffrey, Pereira, Arnaldo
• Primary Examiner(s): Nguyen, Jennifer T
• Application Number:
  US16/138,791
• Publication Number:
  US 20200097076A1
• Time in Patent Office: 613 Days
• Field of Search: 345 7, 345 8, 345156, 345157
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

Embodiments described herein relate to systems, devices, and methods for use in the implementation of a human-computer interface using high-speed, and efficient tracking of user interactions with a User Interface/User Experience that is strategically presented to the user. Embodiments described herein also relate to the implementation of a hardware agnostic human-computer interface that uses neural, oculomotor, and/or electromyography signals to mediate user manipulation of machines and devices.

Claims

1. An apparatus, comprising:

a display configured to present an interactive environment to a user;

an eye-tracker coupled to the display, the eye-tracker including at least two sensors, the at least two sensors being configured to record a plurality of sets of eye-movement signals from an eye of the user, each set of eye-movement signals being recorded, by each sensor from the at least two sensors;

an interfacing device operatively coupled to the display and the eye-tracker, the interfacing device including;

a memory; and

a processor operatively coupled to the memory and configured to;

receive the eye-movement signals from the at least two sensors in the eye-tracker;

generate and present a stimulus, via the interactive environment and via the display, to the user;

compute, based on each set of eye-movement signals from the plurality of sets of eye-movement signals, a set of gaze vectors, each gaze vector from the set of gaze vectors being associated with each sensor from the at least two sensors, the gaze vector associated with each sensor indicating a gaze angle of the eye of the user;

determine a degree of obliqueness of each gaze vector associated with each sensor from the at least two sensors, the degree of obliqueness being relative to a vertical angle associated with that sensor;

determine, based on the degree of obliqueness of each gaze vector associated with each sensor from the at least two sensors, a weight associated with each sensor from the at least two sensors, to generate a set of weights; and

apply the set of weights to the plurality of sets of eye-movement signals to determine a set of calibrated eye-movement signals;

determine, based on the calibrated eye-movement signals, a point of focus of the user;

determine, based on the point of focus of the user, an action intended by the user; and

implement the action intended by the user.

2. The apparatus of claim 1, wherein:

the display includes a display lens configured to project the interactive environment to the user; and

the at least two sensors in the eye-tracker are positioned around the display lens, and along an axes.

3. The apparatus of claim 1, wherein:

the display includes a display lens configured to project the interactive environment to the user; and

the eye-tracker includes at least four sensors, the at least four sensors being positioned around the display lens, and along two orthogonal axes.

4. The apparatus of claim 1, wherein the eye-tracker is further configured to send, to the processor, the eye-movement signals recorded by each sensor from the at least two sensors in an independent manner.

5. The apparatus of claim 1, wherein

the processor is further configured to:

determine, based on an empirically pre-determined weighting function, the weight associated with each sensor from the at least two sensors.

6. The apparatus of claim 1, wherein

the processor is further configured to:

identify a set of missing data points in the plurality of sets of eye-movement signals;

receive, from the eye-tracker, information related to the at least two sensors;

generate, based on the information related to the at least two sensors, a kinematics model of a set of simulated eye-movements of a simulated user;

compute, based on the kinematics model, a plurality of sets of simulated eye-movement signals, each set of simulated eye-movement signals being associated with each sensor from the at least two sensors;

compute a set of replacement data points to replace the set of missing data points in the eye-movement signals received from the at least two sensors, based on the plurality of sets of simulated eye-movement signals; and

incorporate the set of replacement data points to replace the set of missing data points to update the plurality of sets of eye-movement signals associated with the at least two sensors.

7. The apparatus of claim 1, wherein the eye-tracker includes at least four sensors, the four sensors being positioned along two orthogonal axes, and the processor is further configured to compute the set of gaze vectors subtended relative to the two orthogonal axes, the set of gaze vectors configured to collectively represent a gaze angle of the eye of the user.

8. The apparatus of claim 1, wherein the eye-movement signals correspond to a gaze angle of the eye of the user at a first time point, and the processor is further configured to determine, based on the eye-movement signals, a gaze angle of the eye of the user at a second time point different from the first time point, the second time point occurring after the first time point.

9. The apparatus of claim 1, wherein the eye-movement signals correspond to a gaze angle of the eye of the user at a first time point, and are associated with a first measure of momentum at the first time point, and the processor is further configured to determine, based on the eye-movement signals, the gaze angle at the first time point, and the first measure of momentum at the first time point, a gaze angle of the eye of the user at a second time point different from the first time point, the second time point occurring after the first time point.

10. The apparatus of claim 1, further comprising a neural recording device configured to record neural signals generated by the user, the neural signals including electroencephalogram (EEG) signals, wherein the point of focus is a calculated point of focus and the processor is further configured to:

receive the EEG signals, the EEG signals including at least one of visually evoked potentials (VEP), auditory evoked potentials (AEP), motor imagery signals, Event Related Potentials (ERP), and brain state dependent signals;

determine, based on the EEG signals, an expected point of focus of the user;

compute, based on a comparison between the calculated point of focus and the expected point of focus, a measure of error associated with the calculated point of focus;

correct the calculated point of focus, based on the measure of error, to generate a calibrated point of focus of the user.

11. A method, comprising:

presenting, to a user and via a display, a stimulus included in an interactive user interface;

receiving, from an eye-tracker, eye-movement signals associated with the users behavior, the eye-movement signals being recorded independently by at least two sensors positioned on the eye-tracker;

receiving information related to the presented stimulus;

computing based on the eye-movement signals, a gaze vector associated with each sensor from the at least two sensors, the gaze vector indicating a gaze angle of the eye of the user;

determining a degree of obliqueness of the gaze vector associated with each sensor from the at least two sensors, the degree of obliqueness of the gaze vector being relative to a vertical angle associated with that sensor from the at least two sensors;

defining, based on the degree of obliqueness of the gaze vector associated with each sensor from the at least two sensors, a weights associated with each sensor from the at least two sensors, to generate a set of weights;

applying the set of weights to the eye-movement signals recorded independently by each sensor from the at least two sensors, to compute a set of calibrated eye-movement signals, the determining of the point of focus being based on the set of calibrated eye-movement signals;

determining, based on the calibrated eye-movement signals, a point of focus of the user;

determining, based on the point of focus and the stimulus, an action intended by the user; and

implementing the action via the interactive user interface.

12. The method of claim 11, wherein the presenting the stimulus includes projecting an image of the stimulus via a display lens associated with the display.

13. The method of claim 11, wherein the receiving information related to the stimulus includes information related to the properties of the display.

14. The method of claim 11, further comprising:

receiving information related to the at least two sensors, including an indication of positioning of each sensor of the at least two sensors relative to the display, and an indication of positioning of each sensor of the at least two sensors relative to an eye of the user.

15. An apparatus, comprising:

a display configured to present an interactive environment to a user;

an eye-tracker coupled to the display, the eye-tracker including at least two sensors, the at least two sensors being configured to record a plurality of sets of eye-movement signals from an eye of the user, each set of eye-movement signals being recorded, independently, by each sensor from the at least two sensors;

an interfacing device operatively coupled to the display and the eye-tracker, the interfacing device including;

a memory; and

a processor operatively coupled to the memory and configured to;

receive the eye-movement signals from the at least two sensors in the eye-tracker;

generate and present a stimulus, via the interactive environment and via the display, to the user;

compute, based on each sets of eye-movement signals from the plurality of sets of eye-movement signals, a set of gaze vectors, each gaze vector from the set of gaze vectors being associated with each sensor indicating a gaze angle of the eye of the user;

determine a degree of obliqueness of each gaze vector associated with each sensor from the at least two sensors, the degree of obliqueness being relative to a vertical angle associated with that sensor;

determine, based on the degree of obliqueness of each gaze vector associated with each sensor from the at least two sensors and an empirically pre-determined weighting function, a weight associated with each sensor from the at least two sensors, to generate a set of weights;

apply the set of weights to the plurality of sets of eye-movement signals to determine a set of calibrated eye-movement signals;

determine, based on the calibrated eye-movement signals, a point of focus of the user;

determine, based on the point of focus of the user, an action intended by the user; and

implement the action intended by the user.

16. The apparatus of claim 15, wherein:

the display includes a display lens configured to project the interactive environment to the user; and

the at least two sensors in the eye-tracker are positioned around the display lens, and along an axes.

17. The apparatus of claim 15, wherein:

the display includes a display lens configured to project the interactive environment to the user; and

the eye-tracker includes at least four sensors, the at least four sensors being positioned around the display lens, and along two orthogonal axes.

18. The apparatus of claim 15, wherein the eye-tracker includes at least four sensors, the four sensors being positioned along two orthogonal axes, and the processor is further configured to compute, based on the eye-movements signals, a the set of gaze vectors subtended relative to the two orthogonal axes, the set of gaze vectors configured to collectively represent a gaze angle of the eye of the user.

19. The apparatus of claim 15, wherein the eye-movement signals correspond to a gaze angle of the eye of the user at a first time point, and the processor is further configured to determine, based on the eye-movement signals, a gaze angle of the eye of the user at a second time point different from the first time point, the second time point occurring after the first time point.

20. The apparatus of claim 15, wherein the eye-movement signals correspond to a gaze angle of the eye of the user at a first time point, and are associated with a first measure of momentum at the first time point, and the processor is further configured to determine, based on the eye-movement signals, the gaze angle at the first time point, and the first measure of momentum at the first time point, a gaze angle of the eye of the user at a second time point different from the first time point, the second time point occurring after the first time point.

21. The apparatus of claim 15, further comprising:

a neural recording device configured to record neural signals generated by the user, the neural signals including electroencephalogram (EEG) signals, wherein the point of focus is a calculated point of focus and the processor is further configured to;

receive the EEG signals, the EEG signals including at least one of visually evoked potentials (VEP), auditory evoked potentials (AEP), motor imagery signals, Event Related Potentials (ERP), and brain state dependent signals;

determine, based on the EEG signals, an expected point of focus of the user;

compute, based on a comparison between the calculated point of focus and the expected point of focus, a measure of error associated with the calculated point of focus;

correct the calculated point of focus, based on the measure of error, to generate a calibrated point of focus of the user.

22. An apparatus, comprising:

a display configured to present an interactive environment to a user;

an eye-tracker coupled to the display, the eye-tracker including at least two sensors, the at least two sensors being configured to record a plurality of sets of eye-movement signals from an eye of the user, each set of eye-movement signals being recorded, independently, by each sensor from the at least two sensors;

an interfacing device operatively coupled to the display and the eye-tracker, the interfacing device including;

a memory; and

a processor operatively coupled to the memory and configured to;

receive the eye-movement signals from the at least two sensors in the eye-tracker;

generate and present a stimulus, via the interactive environment and via the display, to the user;

identify a set of missing data points in the plurality of sets of eye-movement signals;

receive, from the eye-tracker, information related to the at least two sensors;

generate, based on the information related to the at least two sensors, a kinematics model of a set of simulated eye-movements of a simulated user;

compute, based on the kinematics model, a plurality of sets of simulated eye-movement signals, each set of simulated eye-movement signals being associated with each sensor from the at least two sensors;

compute a set of replacement data points to replace the set of missing data points in the eye-movement signals received from the at least two sensors, based on the plurality of sets of simulated eye-movement signals;

incorporate the set of replacement data points to replace the set of missing data points and to generate calibrated eye-movement signals associated with each sensor from the at least two sensors, the point of focus of the user being determined based on the calibrated eye-movement signals;

determine, based on the calibrated eye-movement signals, a point of focus of the user;

determine, based on the point of focus of the user, an action intended by the user; and

implement the action intended by the user.

23. The apparatus of claim 22, wherein:

the display includes a display lens configured to project the interactive environment to the user; and

the at least two sensors in the eye-tracker are positioned around the display lens, and along an axes.

24. The apparatus of claim 22, wherein:

the display includes a display lens configured to project the interactive environment to the user; and

the eye-tracker includes at least four sensors, the at least four sensors being positioned around the display lens, and along two orthogonal axes.

25. The apparatus of claim 22, wherein the eye-tracker includes at least four sensors, the four sensors being positioned along two orthogonal axes, and the processor is further configured to compute, based on the eye-movements signals, a the set of gaze vectors subtended relative to the two orthogonal axes, the set of gaze vectors configured to collectively represent a gaze angle of the eye of the user.

26. The apparatus of claim 22, wherein the eye-movement signals correspond to a gaze angle of the eye of the user at a first time point, and the processor is further configured to determine, based on the eye-movement signals, a gaze angle of the eye of the user at a second time point different from the first time point, the second time point occurring after the first time point.

27. The apparatus of claim 22, wherein the eye-movement signals correspond to a gaze angle of the eye of the user at a first time point, and are associated with a first measure of momentum at the first time point, and the processor is further configured to determine, based on the eye-movement signals, the gaze angle at the first time point, and the first measure of momentum at the first time point, a gaze angle of the eye of the user at a second time point different from the first time point, the second time point occurring after the first time point.

28. The apparatus of claim 22, further comprising:

a neural recording device configured to record neural signals generated by the user, the neural signals including electroencephalogram (EEG) signals, wherein the point of focus is a calculated point of focus and the processor is further configured to;

receive the EEG signals, the EEG signals including at least one of visually evoked potentials (VEP), auditory evoked potentials (AEP), motor imagery signals, Event Related Potentials (ERP), and brain state dependent signals;

determine, based on the EEG signals, an expected point of focus of the user;

compute, based on a comparison between the calculated point of focus and the expected point of focus, a measure of error associated with the calculated point of focus;

correct the calculated point of focus, based on the measure of error, to generate a calibrated point of focus of the user.

29. A method, comprising:

presenting, to a user and via a display, a stimulus included in an interactive user interface;

receiving, from an eye-tracker, eye-movement signals associated with the users behavior, the eye-movement signals being recorded independently by at least two sensors positioned on the eye-tracker;

receiving information related to the presented stimulus;

computing, based on the eye-movement signals, a set of gaze vectors, of an eye of the user, each gaze vector from the set of gaze vectors being associated with each sensor of the at least four sensors;

resolving the set of gaze vectors along a first axis and a second axis orthogonal to the first axis;

computing a first average gaze vector based on a first weighted average of gaze vectors associated with a first set of sensors, the first set of sensors being grouped along the first axis;

computing a second average gaze vector based on a second weighted average of gaze vectors associated with a second set of sensors, the second set of sensors being grouped along the second axis; and

computing a calibrated gaze vector based on the first average gaze vector and the second average gaze vector;

determining, based on the calibrated gaze vector, a point of focus of the user;

determining, based on the point of focus and the stimulus, an action intended by the user; and

implementing the action via the interactive user interface.

30. The method of claim 29, wherein the presenting the stimulus includes projecting an image of the stimulus via a display lens associated with the display.

31. The method of claim 29, wherein the receiving information related to the stimulus includes information related to the properties of the display.

32. The method of claim 29, further comprising:

receiving information related to the at least two sensors, including an indication of positioning of each sensor of the at least two sensors relative to the display, and an indication of positioning of each sensor of the at least two sensors relative to an eye of the user.