SYSTEM AND METHOD FOR ASSOCIATING MUSIC WITH BRAIN-STATE DATA

• US 20150297109A1
• Filed: 04/22/2015
• Published: 10/22/2015
• Est. Priority Date: 04/22/2014
• Status: Active Grant

• Current Assignee: InteraXon Inc.
• Original Assignee: InteraXon Inc.
• Inventors: GARTEN, ARIEL STEPHANIE, Aimone, Christopher Allen, Vidyarthi, Kapil Jay Mishra, Pino, Locillo (Lou) Giuseppe, Baranowski, Paul Harrison, Rupsingh, Raul Rajiv, Coleman, Trevor, Chabior, Michael Apollo, Ashby, Madeline, Tadich, Paul V., Moffat, Graeme Daniel, Camargo, Javier Arturo Moreno
• Granted Patent: US 10,321,842 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/316 Modalities, i.e. specific diagnostic methods
• CPC Class Codes: A61B 5/372 Analysis of electroencephalograms
• CPC Class Codes: A61B 5/375 using biofeedback
• CPC Class Codes: A61B 5/38 Acoustic or auditory stimuli
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: A61B 5/742 using visual displays A61B5/339 takes precedence
• CPC Class Codes: A61M 2021/0027 by the hearing sense
• CPC Class Codes: A61M 21/00 Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis
• CPC Class Codes: A61M 2205/3375 Acoustical, e.g. ultrasonic, measuring means
• CPC Class Codes: A61M 2205/3561 local, e.g. within room or hospital
• CPC Class Codes: A61M 2205/3584 using modem, internet or bluetooth
• CPC Class Codes: A61M 2205/502 User interfaces, e.g. screens or keyboards
• CPC Class Codes: A61M 2205/52 with memories providing a history of measured variating parameters of apparatus or patient
• CPC Class Codes: A61M 2230/06 Heartbeat rate only
• CPC Class Codes: A61M 2230/10 Electroencephalographic signals
• CPC Class Codes: A61M 2230/42 Rate
• CPC Class Codes: A61M 2230/60 Muscle strain, i.e. measured on the user
• CPC Class Codes: A61M 2230/65 Impedance, e.g. conductivity, capacity

Abstract

A system and method may be provided for associating bio-signal data (e.g. EEG brain scan data) from at least one user with at least one music data item (e.g. song, or piece of music). By associating bio-signal data, or emotions determined therefrom, with music, the system may establish a data store of music associated with emotions. That database may then be leveraged upon determining that a user is feeling a particular emotion through an EEG scan. When a particular emotion is detected in EEG data of a user, the system may then respond based at least partly on the same or similar emotion being associated with one or more music data items in the system. For example, the system may recommend a particular song associated with the same emotion presently being experienced by the user.

Claims

1. An intelligent music system comprising:

(a) at least one bio-signal sensor configured to capture bio-signal sensor data from at least one user;

(b) an input receiver configured to receive music data and the bio-signal sensor data, the music data and the bio-signal sensor data being temporally defined such that the music data corresponds temporally to at least a portion of the bio-signal sensor data;

(c) at least one processor configured to provide;

(i) a music processor to segment the music data into a plurality of time epochs of music, each epoch of music linked to a time stamp;

(ii) a sonic feature extractor to, for each epoch of music, extract a set of sonic features;

(iii) a biological feature extractor to extract, for each epoch of music, a set of biological features from the bio-signal sensor data using the time stamp for the respective epoch of music;

(iv) a metadata extractor to extract metadata from the music data;

(v) a user feature extractor to extract a set of user attributes from the music data and the bio-signal sensor data, the user attributes comprising one or more user actions taken during playback of the music data;

(vi) a machine learning engine to transform the set of sonic features, the set of biological features, the set of metadata, and the set of user attributes into, for each epoch of music, a set of categories that the respective epoch belongs to using one or more predictive models to predict a user reaction of music; and

(d) a music recommendation engine configured to provide at least one music recommendation based on the set of labels or classes.

2. The intelligent music system of claim 1 wherein the input receiver is configured to receive a target emotional state and wherein the system further comprises a music controller to interact with the music recommendation engine to provide at least one music recommendation based on the target emotional state.

3. The intelligent music system of claim 1 wherein the music processor is configured to identify a selection of songs from the music data and add a temporal history to the selection of songs, the temporal history indicating a date and time a user of the at least one users listened to or otherwise selected songs of the selection of songs and an order of the selection of songs, wherein the predictive models use a temporal model for the temporal history the selection of songs.

4. The intelligent music system of claim 3 wherein the selection of songs provides at least a portion of the metadata used for the recommendation.

5. The intelligent music system of claim 1 wherein the at least one processor is configured to provide the one or more predictive models has a learning mode and an operational mode.

6. The intelligent music system of claim 1 wherein each of the categories are nested in a hierarchy of nodes or an ordered list of probabilities of the respective category.

7. The intelligent music system of claim 1 further comprising a plurality of bio-signal sensors configured to capture bio-signal sensor data from a plurality of users and correlate a portion of the bio-signal sensor data across the plurality of users, the portion being mapped to one or more epochs of music, wherein the machine learning engine transform the portion of the bio-signal sensor data across the plurality of users to a common set of categories for use in music recommendations.

8. The intelligent music system of claim 1 further comprising a music effect controller to influence user state by playback or recording of music based the at least one music recommendation.

9. An intelligent music system comprising:

at least one bio-signal sensor comprising at least one brainwave sensor; and

at least one computing device in communication with the least one bio-signal sensor to continuously receive bio-signal data comprising brainwave data of at least one user, the at least one computing device configured to;

define a profile for the at least one user comprising the brainwave data, and user attributes, the brainwave data linked to a timeline;

detect an EEG response as a segment of the brainwave data at a time period on the timeline, the EEG response defining a change in brain state;

correlate the time period to music data to compute a segment of music data corresponding to the segment of the brainwave data of the EEG response;

identify a selection of music data using the segment of music data and the user attributes; and

transmit signals defining a recommendation of a music data item based on the selection of music data.

10. The system of claim 9 wherein at least one computing device is configured to take multiple samples of the brainwave data a different times to detect a plurality of EEG responses and timestamp any detected EEG response.

11. The system of claim 9 wherein the user attributes comprise data fields defining music selections, personality data, and demographic data.

12. The system of claim 9 wherein the EEG response defines a current emotional state of the user, and the selection of music data is linked to a desired emotional state relative to the current emotional state.

13. The system of claim 9 wherein the at least one computing device is configured to receive user feedback to reject or accept the recommendation based on the selection of music data, and refine subsequent selections of music data based on the user feedback.

14. The system of claim 9 wherein the at least one computing device configured to identify the selection of music data by identifying users that have similar EEG responses to the detected EEG response.

15. The system of claim 9 wherein the user attributes comprise data fields defining at least one mental state, and the selection of music data is linked to treatment for the at least one mental state.

16. The system of claim 9 wherein the computing device is configured to determine a correspondence between the received brainwave data and historical data available to the system associated with at least one second user;

and trigger a user correspondence action based at least partly on the determined correspondence.

17. The system of claim 9 wherein the at least one computing device configured to provide at least one digital content item to at least one user at the at least one computing device, determine at least one emotion exhibited by the received brainwave data;

and associate the at least one emotion with the at least one digital content item.

18. The system of claim 9 wherein the at least one bio-signal sensors comprises sensors for receiving data defining physiological measurements of the user.

19. The system of claim 9 further comprising cloud data storage connected to the at least one computing device, the cloud data storage storing the profile, the music data and the brainwave data.

20. The system of claim 9 further comprising an audio input device to receive audio signals corresponding to the music data.

21. The system of claim 9 wherein at least one computing device is configured to generate a data structure with a tag on the music data, the tag defining an emotional state based on the EEG response.

22. The system of claim 9 wherein the EEG response defines a current physical state of the user and wherein the at least one computing device is configured to determine the recommendation based on a desired physical state relative to the current physical state.

23. The system of claim 9 further comprising an interface to a music platform for triggering download or purchase of the music data item of the recommendation.

24. The system of claim 9, further comprising an interface for displaying a current emotional state of the user based on the EEG response.

25. A system comprising:

A plurality of bio-signal sensors, each bio-signal sensor comprising at least one brainwave sensor; and

at least one computing device in communication with the plurality of bio-signals sensor to continuously receive bio-signal data comprising brainwave data of a plurality of users, the at least one computing device configured to;

detect an EEG response as a segment of the brainwave data at a time period;

correlate the time period to music data to compute a segment of music data corresponding to the segment of the brainwave data of the EEG response;

determine a collective emotional state of the plurality of users; and

generate a music data item using the segment of music data and the collective emotional state.