Systems and methods for collecting, analyzing, and sharing bio-signal and non-bio-signal data

• US 10,606,353 B2
• Filed: 09/14/2018
• Issued: 03/31/2020
• Est. Priority Date: 09/14/2012
• Status: Active Grant

• Current Assignee: InteraXon Inc.
• Original Assignee: InteraXon Inc.
• Inventors: Ce Coleman, Trevor, Aimone, Christopher Allen, Garten, Ariel Stephanie, Pino, Locillo (Lou) Giuseppe, Baranowski, Paul Harrison, Rupsingh, Raul Rajiv, Vidyarthi, Kapil Jay Mishra, Moffat, Graeme, Mackenzie, Samuel Thomas
• Primary Examiner(s): Holcomb, Mark
• Application Number:
  US16/132,242
• Publication Number:
  US 20190113973A1
• Time in Patent Office: 564 Days
• Field of Search: 705 2- 4
• CPC Class Codes: A61B 5/0006 ECG or EEG signals
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0024 for multiple sensor units attached to the patient, e.g. using a body or personal area network
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/02438 with portable devices, e.g. worn by the patient
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/369 Electroencephalography [EEG] devices for psychotechnics A61B5/16
• CPC Class Codes: A61B 5/4088 Diagnosing of monitoring cognitive diseases, e.g. Alzheimer, prion diseases or dementia
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/126 Evolutionary algorithms, e.g. genetic algorithms or genetic programming
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G16H 40/63 : for local operation
• CPC Class Codes: G16H 40/67 : for remote operation
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 50/70 : for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: G16Z 99/00 : Subject matter not provided for in other main groups of this subclass
• CPC Class Codes: H04L 12/16 : Arrangements for providing special services to substations
• CPC Class Codes: H04L 12/1813 : for computer conferences, e.g. chat rooms instant messaging H04L51/04; protocols for multimedia communication H04L65/1101; arrangements for multi-party communication H04L65/403; telephonic conference arrangements H04M3/56; television conference systems H04N7/15
• CPC Class Codes: H04L 67/01 : Protocols
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

A computer network implemented system for improving the operation of one or more biofeedback computer systems is provided. The system includes an intelligent bio-signal processing system that is operable to: capture bio-signal data and in addition optionally non-bio-signal data; and analyze the bio-signal data and non-bio-signal data, if any, so as to: extract one or more features related to at least one individual interacting with the biofeedback computer system; classify the individual based on the features by establishing one or more brain wave interaction profiles for the individual for improving the interaction of the individual with the one or more biofeedback computer systems, and initiate the storage of the brain waive interaction profiles to a database; and access one or more machine learning components or processes for further improving the interaction of the individual with the one or more biofeedback computer systems by updating automatically the brain wave interaction profiles based on detecting one or more defined interactions between the individual and the one or more of the biofeedback computer systems. A number of additional system and computer implemented method features are also provided.

Claims

1. A brainwave monitoring system comprising:

a plurality of client computing devices, each of the plurality of client computing devices in communication with at least one bio-signal sensor including at least one electroencephalography (EEG) bio-signal sensor; and

at least one computer server in communication with the plurality of computing devices over a communications network, the at least one computer server configured to;

receive time-coded EEG bio-signal data from at least one of the plurality of client computing devices, the time-coded EEG bio-signal data having metadata indicating a user identifier of the plurality of user identifiers;

acquire time-coded feature event data;

generate a cloud pipeline instance to process the time-coded EEG bio-signal data and the time coded feature event data, the cloud pipeline instance defining pipeline parameters for a classification model;

extract, using the cloud pipeline instance, feature events from the time coded feature event data at feature event time codes, each feature event being a set of variables and corresponding values at at least one feature event time code;

automatically search the feature events to identify a pattern, the pattern linked to a feature event time code of the at least one feature event time code, the pattern representing user response associated with the feature event data at the feature event time code;

using the feature event time code linked to the pattern identified in the feature-events, label segments in the time-coded EEG bio-signal data having EEG bio-signal time-codes being same or similar to the feature event time code linked to the pattern;

update the classification model with the EEG bio-signal features extracted from labelled segments of the time-coded EEG bio-signal data, the classification model for predicting brain state based on the pipeline parameters;

determine a response classification of the segments of the time-coded EEG bio-signal data using the classification model and pipeline parameters, the response classification being an automatic prediction of a brain state at the EEG bio-signal time codes;

update, for the user identifier, a bio-signal interaction profile based on the classification model, the response classification, the time-coded EEG bio-signal data associated with that user identifier, and the time-coded feature event data;

receive additional time-coded EEG bio-signal data associated with the user identifier; and

generate an encryption key using the bio-signal interaction profile associated with the user identifier and segments of the additional time-coded EEG bio-signal data associated with the user identifier.

2. The system of claim 1, wherein the encryption key is unique to the user identifier.

3. The system of claim 1, wherein the additional time-coded EEG bio-signal data is associated with a set of discrete bins, each of the discrete bins having value boundaries, and the encryption key is generated based at least in part on the association between the additional time-coded EEG bio-signal data and the set of discrete bins.

4. The system of claim 1, wherein the bio-signal interaction profile is a brain model associated with the user identifier.

5. The system of claim 1, wherein the additional time-coded EEG bio-signal data is associated with a task prompting a user response.

6. The system of claim 1, wherein the at least one computer server is further configured to encrypt data using the encryption key.

7. The system of claim 6, wherein the at least one computer server is further configured to discard the encryption after the data is encrypted.

8. The system of claim 1, wherein the additional time-coded EEG bio-signal data is associated with a first point in time, and the at least one computer server is further configured to:

receive a second additional time-coded EEG bio-signal data associated with the user identifier at a second point in time; and

identify a variance of values of the additional time-coded EEG bio-signal data and the second additional time-coded EEG bio-signal data,

generate a second encryption key using the bio-signal interaction profile associated with the user identifier, segments of the second additional time-coded EEG bio-signal data, and the variance.

9. The system of claim 8, wherein the variance represents a change in a user response to a repeated task at the first point in time and the second point in time.

10. The system of claim 1, further comprising at least one non-bio-signal sensor at and in communication with each of the client computing devices, the at least one computer server further configured to:

receive time-coded non-bio-signal sensor data from the at least one non-bio-signal sensor;

wherein the generated encryption key is based at least partly on the received time-coded non-bio-signal sensor data.

11. A brainwave monitoring system comprising:

at least one user effector to provide a real-time biofeedback output; and

at least one computer server in communication with a plurality of computing devices over a communications network, each of the plurality of client computing devices in communication with at least one bio-signal sensor including at least one electroencephalography (EEG) bio-signal sensor, the at least one computer server configured to;

receive time-coded EEG bio-signal data from each of the plurality of client computing devices, the time-coded EEG bio-signal data associated with a plurality of user identifiers;

acquire time-coded feature event data;

generate a cloud pipeline instance to process the time-coded EEG bio-signal data and the time coded feature event data, the cloud pipeline instance defining pipeline parameters for a classification model;

extract, using the cloud pipeline instance, feature events from the time coded feature event data at feature event time codes, each feature event being a set of variables and corresponding values at at least one feature event time code;

automatically search the feature events to identify patterns that are statistically significant, each pattern linked to a feature event time code of the at least one feature event time code, the pattern representing user response associated with the feature event data at the feature event time code;

using the feature event time codes linked to the patterns identified in the feature-events, label segments in the time-coded EEG bio-signal data having EEG bio-signal time-codes being same or similar to the feature event time codes linked to the patterns;

update the classification model with the EEG bio-signal features extracted from labelled segments of the time-coded EEG bio-signal data, the classification model for predicting brain state based on the pipeline parameters;

determine a response classification of the segments of the time-coded EEG bio-signal data using the classification model and pipeline parameters, the response classification being an automatic prediction of a brain state at the EEG bio-signal time codes;

generate the real-time biofeedback output based on segments of the time-coded EEG bio-signal data, the classification model and the response classification;

the at least one user effector configured to provide the real-time biofeedback output for each of the plurality of user identifiers.

12. The system of claim 11, wherein the real-time biofeedback output is an aggregation of segments of the EEG bio-signal data from each user identifier.

13. The system of claim 11, wherein the real-time biofeedback output is a modulation of some of the EEG bio-signal data from each user identifier.

14. The system of claim 11, wherein the at least one effector is associated with each user identifier and are co-located.

15. The system of claim 11, wherein the at least one effector is associated with each user identifier and are remote from each other.

16. The system of claim 11, wherein the at least one computer server is configured to send the real-time biofeedback output to an audio and visual display that is common to the users associated with the plurality of user identifiers.

17. The system of claim 11, wherein the at least one computer server is configured to generate, for each of the plurality of user identifiers, an individual real-time biofeedback output based on segments of the time-coded EEG bio-signal data, the classification model and the response classification, the at least one user effector configured to provide the individual real-time biofeedback output for that user identifier.

18. The system of claim 11, wherein the real-time biofeedback output comprises at least one of visual data, audio data and tactile data.

19. The system of claim 11 wherein the at least one user effector is part of a augment reality system and the real-time biofeedback output comprises visual elements on a display that is part of the augment reality system.

20. The system of claim 11 wherein the at least one user effector comprises a visual indication on a display.