BIO-SENSING AND EYE-TRACKING SYSTEM

• US 20190174039A1
• Filed: 08/15/2017
• Published: 06/06/2019
• Est. Priority Date: 08/15/2016
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Siddharth Siddharth, Terrence Sejnowski, Tzyy-Ping Jung, Regents of the University of California (University of California)
• Inventors: Jung, Tzyy-Ping, Siddharth, Siddharth, Sejnowski, Terrence
• Granted Patent: US 11,032,457 B2
• Time in Patent Office: Days
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Methods, systems, and devices are disclosed for implementing a low-cost wearable multi-modal bio-sensing system capable of recording bio-markers and eye-gaze overlaid on world view in real-world settings. In an exemplary embodiment, a bio-sensing system uses at least two cameras and one or more bio-sensors to record a variety of events and bio-marker data. In another exemplary embodiment, the recorded information is used to track the eye position, calculate the pupil dimensions, and calculate the gaze of a human being. The eye position and dimensions of the pupil can be used for emotion recognition.

Claims

1. A bio-sensing system, comprising:

a frame structured to be placed on a user'"'"'s face;

a first camera coupled to the frame and facing towards an eye of the user to capture a first set of images of the eye;

a second camera coupled to the frame and facing away from the user and configured to capture a second set of images of an environment from the user'"'"'s perspective;

one or more sensors configured to measure biological functions of the user and to generate sensor data; and

a computer electrically coupled to the one or more sensors, the first camera and the second camera.

2. The bio-sensing system of claim 1, wherein the computer includes at least one processor in communication with a memory operable to execute instructions to cause the computer to perform a method comprising:

receiving the first set of images from the first camera;

detecting location of a pupil of the eye based on the received first set of images; and

calculating coordinates of the pupil and a diameter of the pupil based on the received first set of images.

3. The bio-sensing system of claim 2, wherein the at least one processor is operable to execute instructions to further cause the computer to perform the method comprising:

calculating a confidence of the pupil being present in the first set of images.

4. The bio-sensing system of claim 2, wherein the at least one processor is operable to execute instructions to further cause the computer to perform the method comprising:

receiving the second set of images from the second camera;

detecting one or more objects based on the received second set of images;

transmitting the coordinates of the pupil and the second set of images to a remote computer; and

transmitting coordinates of the detected one or more objects to the remote computer, wherein the remote computer includes at least one second processor in communication with a second memory operable to execute instructions to cause the remote computer to perform a method comprising;

highlighting on a display the detected one or more objects.

5. The bio-sensing system of claim 4, wherein the at least one processor is operable to execute instructions to further cause the computer to perform a method comprising:

determining user excitement by measuring a rate of increase of the diameter of the pupil

7. The bio-sensing system of claim 4, wherein the remote computer includes at least one second processor in communication with the second memory operable to execute instructions to cause the remote computer to perform a method comprising:

receiving the coordinates of the pupil and the second set of images;

prompting the user to look at one or more locations;

calibrating a gaze of the user based on the calculated coordinates of the pupil and based on the user'"'"'s pupil response to the prompting to look at the one or more locations; and

overlaying a position of the pupil onto the second set of images.

8. The bio-sensing system of claim 1, wherein

the one or more sensors comprises a photoplethysmogram (PPG) sensor and an accelerometer, wherein

the PPG sensor and the accelerometer are structured to be attached to an earlobe of the user,

the PPG sensor is configured to generate PPG sensor data by capturing infrared light reflected from the earlobe of the user,

the accelerometer is configured to generate accelerometer data by detecting movement of the user;

the at least one processor is operable to execute instructions to further cause the computer to perform the method comprising;

filtering the PPG sensor data;

filtering the accelerometer data;

converting the filtered PPG sensor data to digital PPG sensor data;

converting the filtered accelerometer data to digital accelerometer data;

estimating, based on the digital accelerometer data, noise contributed from user movement; and

removing from the digital PPG sensor data the noise contributed from user movement to obtain a noise filtered PPG sensor data.

12. The bio-sensing system of claim 8, wherein

the one or more sensors further comprises an electroencephalogram (EEG) sensor, wherein the EEG sensor is configured to generate an EEG signal based on electrical activity of a brain of the user;

the at least one processor is operable to execute instructions to further cause the computer to perform the method comprising;

filtering the EEG signal to remove noise;

performing fast Fourier transform (FFT) on the filtered EEG signal to obtain an FFT EEG signal;

calculating biomarkers from the FFT EEG signal;

calculating heart rate based on the noise filtered PPG sensor data; and

measuring heart rate variability based on the noise filtered PPG sensor data.

13. The bio-sensing system of claim 12, wherein the at least one processor is operable to execute instructions to further cause the computer to perform the method comprising:

recognizing an emotion of the user based on the determined user excitement, the calculated heart rate, the measured heart rate variability, and the calculated biomarkers; and

tagging the emotion of the user associated with the object seen by the user.

14. The bio-sensing system of claim 13, wherein the tagging of the emotion with the object includes recording a time when the user saw the object.

17. The bio-sensing system of claim 1, wherein the first camera is an infrared camera, and wherein the second camera includes a wide-angle lens.

6. (canceled)

9. (canceled)

10. (canceled)

11. (canceled)

15. (canceled)

16. (canceled)

18. (canceled)

19. (canceled)

20. (canceled)

21. A method of operating a bio-sensing system, the method comprising:

receiving, by a first computer, a first set of images from a first camera;

detecting, by the first computer, a location of a pupil of an eye based on the received first set of images; and

calculating, by the first computer, coordinates of the pupil and a diameter of the pupil based on the received first set of images;

receiving, by the first computer, a second set of images from a second camera; and

transmitting, by the first computer, the coordinates of the pupil and the second set of images to a remote computer.

22. The method of claim 21, further comprising:

calculating a confidence of the pupil being present in the first set of images.

23. The method of claim 21, further comprising:

determining user excitement by measuring a rate of increase of the diameter of the pupil.

24. The method of claim 21, further comprising:

detecting, by the first computer, one or more objects based on the received second set of images; and

transmitting coordinates of the detected one or more objects to the remote computer.

26. The method of claim 21, further comprising:

filtering PPG sensor data received from a photoplethysmogram (PPG) sensor;

filtering accelerometer data received from an accelerometer;

converting the filtered PPG sensor data to digital PPG sensor data;

converting the filtered accelerometer data to digital accelerometer data;

estimating, based on the digital accelerometer data, noise contributed from user movement; and

removing from the digital PPG sensor data the noise contributed from user movement to obtain a noise filtered PPG sensor data.

29. The method of claim 26, further comprising:

receiving an electroencephalogram (EEG) signal from an EEG sensor;

filtering the EEG signal to remove noise;

performing fast Fourier transform (FFT) on the filtered EEG signal to obtain an FFT EEG signal;

calculating biomarkers from the FFT EEG signal;

calculating heart rate based on the noise filtered PPG sensor data; and

measuring heart rate variability based on the noise filtered PPG sensor data.

30. The method of claim 29, further comprising:

recognizing an emotion of the user based on the determined user excitement, the calculated heart rate, the measured heart rate variability, and the calculated biomarkers; and

tagging the emotion of the user associated with the object seen by the user.

31. The method of claim 30, wherein the tagging of the emotion with the object includes recording a time when the user saw the object.

34. The method of claim 21, wherein the first camera is an infrared camera, and wherein the second camera includes a wide-angle lens.

25. (canceled)

27. (canceled)

28. (canceled)

32. (canceled)

33. (canceled)

35. (canceled)