Wearable computing apparatus and method

• US 10,365,716 B2
• Filed: 03/17/2014
• Issued: 07/30/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: InteraXon Inc.
• Original Assignee: InteraXon Inc.
• Inventors: Aimone, Christopher Allen, Garten, Ariel Stephanie, Coleman, Trevor, Pino, Locillo (Lou) Giuseppe, Vidyarthi, Kapil Jay Mishra, Baranowski, Paul Harrison, Chabior, Michael Apollo, Chong, Tracy, Rupsingh, Raul Rajiv, Ashby, Madeline, Tadich, Paul V.
• Primary Examiner(s): Mandeville, Jason M
• Application Number:
  US14/216,925
• Publication Number:
  US 20140347265A1
• Time in Patent Office: 1,961 Days
• Field of Search: 345156
• CPC Class Codes: A61B 2560/0493 controlled by voice
• CPC Class Codes: A61B 5/0024 for multiple sensor units attached to the patient, e.g. using a body or personal area network
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/02438 with portable devices, e.g. worn by the patient
• CPC Class Codes: A61B 5/0261 using optical means, e.g. infrared light
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/375 using biofeedback
• CPC Class Codes: A61B 5/378 Visual stimuli
• CPC Class Codes: A61B 5/38 Acoustic or auditory stimuli
• CPC Class Codes: A61B 5/389 Electromyography [EMG]
• CPC Class Codes: A61B 5/398 Electrooculography [EOG], e.g. detecting nystagmus; Electroretinography [ERG]
• CPC Class Codes: A61B 5/4064 Evaluating the brain A61B5/031, A61B5/369, A61B5/14553 take precedence
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A61B 5/744 Displaying an avatar, e.g. an animated cartoon character
• CPC Class Codes: A61M 2021/0022 by the tactile sense, e.g. vibrations
• CPC Class Codes: A61M 2021/0027 by the hearing sense
• CPC Class Codes: A61M 2021/0044 by the sight sense
• CPC Class Codes: A61M 2021/0066 with heating or cooling
• CPC Class Codes: A61M 21/00 : Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis
• CPC Class Codes: A61M 2230/10 : Electroencephalographic signals
• CPC Class Codes: A61M 2230/14 : Electro-oculogram [EOG]
• CPC Class Codes: G02B 2027/0187 : slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 : Head mounted
• CPC Class Codes: G02C 11/10 : Electronic devices other than hearing aids
• CPC Class Codes: G02C 7/027 : considering wearer's parameters
• CPC Class Codes: G06F 16/90 : Details of database functions independent of the retrieved data types
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 : Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/048 : Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0487 : using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G09G 3/003 : to produce spatial visual effects
• CPC Class Codes: G16H 20/70 : relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/67 : for remote operation
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04M 1/05 : specially adapted for use on head, throat or breast
• CPC Class Codes: H04M 2250/12 : including a sensor for measuring a physical value, e.g. temperature or motion
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services
• CPC Class Codes: H04W 4/30 : Services specially adapted for particular environments, situations or purposes

Abstract

A method is provided, performed by a wearable computing device comprising at least one bio-signal measuring sensor, the at least one bio-signal measuring sensor including at least one brainwave sensor, comprising: acquiring at least one bio-signal measurement from a user using the at least one bio-signal measuring sensor, the at least one bio-signal measurement comprising at least one brainwave state measurement; processing the at least one bio-signal measurement, including at least the at least one brainwave state measurement, in accordance with a profile associated with the user; determining a correspondence between the processed at least one bio-signal measurement and at least one predefined device control action; and in accordance with the correspondence determination, controlling operation of at least one component of the wearable computing device, such as modifying content displayed on a display of the wearable computing device. Various types of bio-signals, including brainwaves, may be measured and used to control the device in various ways.

Claims

1. A method, performed by a wearable eyeglass frame comprising at least one bio-signal measuring sensor and a display, the at least one bio-signal measuring sensor including at least one electrophysiological sensor, the at least one electrophysiological sensor including at least one EOG sensor and at least one EEG sensor, comprising:

displaying at least one item on the display;

acquiring bio-signal data from a user using the at least one bio-signal measuring sensor, the bio-signal data comprising electrophysiological data from the at least one electrophysiological sensor;

processing the bio-signal data, in accordance with a profile associated with the user to compute at least one brainwave state measurement and at least one eye activity measurement, the at least one eye activity measurement computed using the electrophysiological data from the at least one EOG sensor, the at least one eye activity measurement based on changes in an electrical potential of the bio-signal data from the at least one EOG sensor, wherein the processing comprises detecting, within the at least one brainwave state measurement, an event related potential associated with the display of the at least one item;

determining, using the at least one eye activity measurement, positions and times of eye movements between points of focus on elements in a scene, the scene including the display, and positions and times of eye gaze fixations on points of focus on elements in the scene;

identifying a pattern of saccadic eye movement based on the eye movements and the eye gaze fixations;

upon identification of the pattern, associating the pattern with a condition of saliences of elements of the scene to the user;

determining a correspondence between the at least one brainwave state measurement, the eye movements, the eye gaze fixations, the saliences of elements of the scene to the user, and at least one predefined device control action; and

in accordance with the determined correspondence, controlling operation of at least one component of the eyeglass frame by modifying or initiating the modification of an image displayed on the display.

2. The method of claim 1 further comprising establishing the profile by calibration using a pre-determined set of exercises and acquiring baseline bio-signal data from the user using the at least one bio-signal measuring sensor.

3. The method of claim 1 further comprising establishing the profile using historical bio-signal data from the user.

4. The method of claim 1 wherein the eye activity measurement is based on at least one of EOG, EEG, event related potentials, and steady state visual evoked potential.

5. The method of claim 1 further comprising:

displaying an application interface on the display with at least one indicator;

detecting a selection of the at least one indicator using the at least one brainwave state measurement, the eye movements and the eye gaze fixations;

upon detecting the selection, activating a memory method for the application interface, the memory method comprising a set of electronic interactions to attempt to stimulate the user and aid the user in locating visual information based on items that the user has seen before on the display.

6. The method of claim 1 further comprising:

displaying an application interface on the display with at least one indicator;

detecting a virtual selection of the at least one indicator using the at least one brainwave state measurement, the eye movements and the eye gaze fixations;

upon detecting the selection, activating an inspiration method for the application interface, the inspiration method comprising a set of electronic interactions to attempt to stimulate the user and aid the user in locating visual information based on items that the user has not seen before on the display.

7. The method of claim 1 further comprising:

generating the profile associated with the user by configuring settings and pipeline parameters tuned to optimize the user'"'"'s experience.

8. The method of claim 1 further comprising:

generating an instance of a pipeline with pipeline parameters;

processing the bio-signal data using the instance of the pipeline with the pipeline parameters;

updating the profile associated with the user with the pipeline parameters.

9. The method of claim 1 further comprising:

determining a level of user interest based on the at least one brainwave state measurement, the eye movements and the eye fixations.

10. The method of claim 1 wherein each of the at least one item flashes, independently, at a respective predetermined frequency, wherein the processing comprises detecting, within the at least one brainwave state measurement, for a steady state visually evoked potential having a frequency associated with any of the respective predetermined frequencies.

11. The method of claim 1 further comprising displaying an application interface on the display with at least one indicator, wherein the determining of the correspondence between the at least one brainwave state measurement, the eye movements, the eye gaze fixations, the saliences of elements of the scene to the user, and at least one predefined device control action includes:

detecting a selection of the at least one indicator using the at least one brainwave state measurement and a fixation in the eye gaze fixations.

12. The method of claim 1, wherein the eyeglass frame further comprises at least one orientation sensor selected from an accelerometer, a gyroscopic sensor, and a compass;

wherein the method further comprises determining a head orientation using the at least one orientation sensor; and

wherein the determining a correspondence between the at least one brainwave state measurement, the eye movements, the eye gaze fixations, the saliences of elements of the scene to the user and the at least one predefined device control action further comprises determining a correspondence between the at least one brainwave state measurement, the eye movements, the eye gaze fixations, the saliences of elements of the scene to the user, the head orientation, and the at least one predefined device control action.

13. A non-transitory computer program product tangibly embodying code that, when executed by a processor, causes the processor to:

display at least one object on a display of a wearable eyeglass frame;

acquire bio-signal data from a user using at least one bio-signal measuring sensor, the at least one bio-signal measuring sensor including at least one electrophysiological sensor, the at least one electrophysiological sensor including at least one EOG sensor and at least one EEG sensor, the bio-signal data comprising electrophysiological data from the at least one electrophysiological sensor;

process the bio-signal data in accordance with a profile associated with the user to compute at least one brainwave state measurement and at least one eye activity measurement, the at least one eye activity measurement computed using the electrophysiological data from the at least one EOG sensor, the at least one eye activity measurement based on changes in an electrical potential of the bio-signal data from the at least one EOG sensor, wherein the processing comprises detecting, within the at least one brainwave state measurement, an event related potential associated with the display of the at least one object;

determine, using the at least one eye activity measurement, positions and times of eye movements between points of focus on elements in a scene, the scene including the display, and positions and times of eye gaze fixations on points of focus on elements in the scene;

identify a pattern of saccadic eye movement based on the eye movements and the eye gaze fixations;

upon identification of the pattern, associate the pattern with a condition of saliences of elements of the scene to the user;

determine a correspondence between the at least one brainwave state measurement, the eye movements, the eye gaze fixations, the saliences of elements of the scene to the user, and at least one predefined device control action; and

in accordance with the determined correspondence, control operation of at least one component of the wearable eyeglass frame by modifying or initiating the modification of an image displayed on the display.

14. A wearable eyeglass frame comprising:

at least one bio-signal measuring sensor, the at least one bio-signal measuring sensor including at least one electrophysiological sensor, the at least one electrophysiological sensor including at least one EOG sensor and at least one EEG sensor;

a display;

at least one data processor coupled to the at least one bio-signal measuring sensor; and

a non-transitory computer-readable medium or media comprising computer-executable instructions configured to cause the at least one data processor to;

display at least one item on the display;

acquire bio-signal data from a user using the at least one bio-signal measuring sensor, the bio-signal data comprising electrophysiological data from the at least one electrophysiological sensor;

process the bio-signal data in accordance with a profile associated with the user to compute at least one brainwave state measurement and at least one eye activity measurement, the at least one eye activity measurement computed using the electrophysiological data from the at least one EOG sensor, the at least one eye activity measurement based on changes in an electrical potential of the bio-signal data from the at least one EOG sensor, wherein the processing comprises detecting, within the at least one brainwave state measurement, an event related potential associated with the display of the at least one item;

determine, using the at least one eye activity measurement, positions and times of eye movements between points of focus on elements in a scene, the scene including the display, and positions and times of eye gaze fixations on points of focus on elements in the scene;

identify a pattern of saccadic eye movement based on the eye movements and the eye gaze fixations;

upon identification of the pattern, associate the pattern with a condition of saliences of elements of the scene to the user;

determine a correspondence between the at least one brainwave state measurement, the eye movements, the eye gaze fixations, the saliences of elements of the scene to the user, and at least one predefined device control action; and

in accordance with the determined correspondence, control operation of at least one component of the eyeglass frame by modifying or initiating the modification of an image displayed on the display.

15. The eyeglass frame of claim 14 further comprising at least one eye-tracking sensor, the non-transitory computer-readable medium or media comprising the computer-executable instructions configured to cause the at least one data processor to:

acquire at least one eye-tracking measurement from the user using the at least one eye-tracking sensor, the at least one eye-tracking measurement including at least one of an eye blink frequency and an eye blink duration;

process the at least one bio-signal data, the eye-tracking measurement and the displayed location of one of the at least one displayed item; and

in accordance with the determined at least one predefined device control action, control operation of the at least one display by modifying or initiating the modification of at least one displayed item.