Mobile user borne brain activity data and surrounding environment data correlation system

• US 9,451,899 B2
• Filed: 06/30/2015
• Issued: 09/27/2016
• Est. Priority Date: 02/15/2006
• Status: Active Grant

• Current Assignee: Virtual Video Reality By Ritchey Limited Liability Corporation
• Original Assignee: Virtual Video Reality By Ritchey LLC
• Inventors: Ritchey, Kurtis John, Ritchey, Kenneth Ira
• Primary Examiner(s): Harrell, Robert B
• Application Number:
  US14/788,437
• Publication Number:
  US 20150324692A1
• Time in Patent Office: 455 Days
• Field of Search: 348/36
• US Class Current: 1/1
• CPC Class Codes: A61B 2560/0242 adapted to measure environmental factors, e.g. temperature, pollution
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0024 for multiple sensor units attached to the patient, e.g. using a body or personal area network
• CPC Class Codes: A61B 5/0059 using light, e.g. diagnosis by transillumination, diascopy, fluorescence A61B5/0093, A61B5/1455, A61B5/02416 and A61B5/0261 take precedence
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/245 specially adapted for magnetoencephalographic [MEG] signals
• CPC Class Codes: A61B 5/369 Electroencephalography [EEG] devices for psychotechnics A61B5/16
• CPC Class Codes: A61B 5/37 Intracranial electroencephalography [IC-EEG], e.g. electrocorticography [ECoG]
• CPC Class Codes: A61B 5/372 Analysis of electroencephalograms
• CPC Class Codes: A61B 5/384 Recording apparatus or displays specially adapted therefor
• CPC Class Codes: A61B 5/4064 Evaluating the brain A61B5/031, A61B5/369, A61B5/14553 take precedence
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/686 Permanently implanted devices, e.g. pacemakers, other stimulators, biochips A61B5/6861 takes precedence
• CPC Class Codes: A61B 5/7246 using correlation, e.g. template matching or determination of similarity
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/745 using a holographic display
• CPC Class Codes: A61B 6/501 for diagnosis of the head, e.g. neuroimaging or craniography
• CPC Class Codes: G01R 33/4806 Functional imaging of brain activation
• CPC Class Codes: G02B 13/06 Panoramic objectives; So-called "sky lenses" including panoramic objectives having reflecting surfaces
• CPC Class Codes: G02B 15/10 : by adding a part, e.g. close-up attachment
• CPC Class Codes: G02B 2027/0138 : comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 : comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 : slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 : Head mounted
• CPC Class Codes: G03B 37/00 : Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
• CPC Class Codes: G03H 1/2294 : Addressing the hologram to an active spatial light modulator
• CPC Class Codes: G05D 1/0016 : characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 : by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G06F 1/1626 : with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 : the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 16/90 : Details of database functions independent of the retrieved data types
• CPC Class Codes: G06F 16/90335 : Query processing
• CPC Class Codes: G06F 2203/0381 : Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 : Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 : Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/147 : using display panels
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G09G 2370/16 : Use of wireless transmission of display information
• CPC Class Codes: G09G 2380/08 : Biomedical applications
• CPC Class Codes: G09G 3/003 : to produce spatial visual effects
• CPC Class Codes: G09G 3/03 : specially adapted for displays having non-planar surfaces, e.g. curved displays
• CPC Class Codes: G09G 3/035 : for flexible display surfaces
• CPC Class Codes: G09G 5/026 : Control of mixing and/or overlay of colours in general G09G5/022 and G09G5/024 take precedence
• CPC Class Codes: G09G 5/14 : Display of multiple viewports
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G16H 40/63 : for local operation
• CPC Class Codes: H04N 23/58 : Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/00 : Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

A mobile user borne brain activity data and surrounding environment data correlation system comprising a brain activity sensing subsystem, a recording subsystem, a measurement computer subsystem, a user sensing subsystem, a surrounding environment sensing subsystem, a correlation subsystem, a user portable electronic device, a non-transitory computer readable medium, and a computer processing device. The mobile user borne system collects and records brain activity data and surrounding environment data and statistically correlates and processes the data for communicating the data into a recipient biological, mechanical, or bio-mechanical system.

Claims

1. A user borne system comprising:

a brain activity sensing subsystem configured to collect data corresponding to brain activity of a user;

a measurement computer subsystem configured to generate data representing quantifications of perceptions of said user;

a user sensing subsystem configured to collect data corresponding to user events;

a surrounding environment sensing subsystem configured to collect data corresponding to the user'"'"'s surrounding environment;

a recording subsystem configured to record said data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, and surrounding environment sensing subsystem;

a correlation subsystem configured to create correlation subsystem data comprising relationships between said data corresponding to said brain activity of said user and said data corresponding to said user events and surrounding environment;

a user mobile electronic device in communication with said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, and recording subsystem, said user mobile electronic device including an interactive graphic user interface and being configured to;

operate as a host computer processing subsystem for command, control, and processing of signals to and from said brain activity sensing subsystem, user sensing subsystem, surrounding environment sensing subsystem, and correlation subsystem;

command said brain activity sensing subsystem to transmit brain activity and pattern data to said correlation subsystem; and

command said user sensing subsystem and surrounding environment sensing subsystem to transmit processed sensor data to said correlation subsystem,

said correlation subsystem being configured to receive and perform correlation processing operations to determine an extent of neural relationships between data received from said user mobile electronic device and said brain activity sensing subsystem, user sensing subsystem, and surrounding environment sensing subsystem to derive neural correlates of consciousness of conscious precepts of said user;

a non-transitory computer readable medium configured to store data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem for performing queries on real-time and near real-time data received from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem for determining whether to keep or disregard said data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem based on pre-established rule-sets and user interactive command and control from said user mobile electronic device; and

a computer processing device configured to process and communicate at least a portion of said data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem into at least one of a recipient biological, mechanical, or bio-mechanical system.

2. The user borne system according to claim 1, wherein said user borne system and recipient biological, mechanical, or bio-mechanical system include at least one of intrusion detection software, hardware, or firmware application and information security software, hardware, or firmware application that provides at least one of firewall protection, virus protection, privacy protection, or user authentication capabilities.

3. The user borne system according to claim 1, wherein said user borne system and recipient biological, mechanical, or bio-mechanical system further comprise at least one remote sensing technology that measures distance by illuminating a target with a laser and analyzing light reflected by said target for at least one of remote sensing, contour mapping, or aiding in determining distance to said target in said surrounding environment.

4. The user borne system according to claim 1, wherein said brain activity sensing subsystem is configured to record signatures and derive measurements on at least one of a whole, region, neural, or electro-chemical interaction in the synaptic cleft of the user'"'"'s brain at the molecular level in order to identify neural correlates of consciousness such that various components of perception across the user'"'"'s brain that form a totality of a conscious precept identify a minimal set of components of neural material, chemical, electrical, and associated activity that define a thought or memory of the conscious precept in the user'"'"'s mind or said surrounding environment, said correlation subsystem being configured to correlate and filter said components of perception to create a relational database for input into an artificial neural network that at least one mimics, supplements, or enhances the user'"'"'s brain or said recipient biological, mechanical, or biomechanical system.

5. The user borne system according to claim 1, wherein said brain activity sensing subsystem includes at least one of a neuro-stimulation fiber optic light emitter or neuro-stimulation micro-electrode for at least one of diagnostic purposes, performance enhancement, or detecting performance degradation of functions of said user'"'"'s brain.

6. The user borne system according to claim 1, wherein said user borne system is designed to blend into the user'"'"'s natural appearance by incorporating at least one of a prosthetic device with human skin color and shape, grafted skin, synthetic skin, a display, a fashion accessory, body art, hair piece, skull cap, jewelry, cap, hat, material covering, or clothing.

7. The user borne system according to claim 1, wherein said recipient biological, mechanical, or bio-mechanical system looks substantially like a human.

8. The user borne system according to claim 1, wherein said recipient biological, mechanical, or bio-mechanical is configured to act substantially like a human.

9. The user borne system according to claim 1, wherein said surrounding environment sensing subsystem data comprises video recording play back capability for playing back video derived from brain activity signatures for comparison of real world recorded imagery versus brain signal motion imagery of the user.

10. The user borne system according to claim 1, wherein at least a portion of said user borne system is configured to be supported by an exoskeleton worn by said user.

11. The user borne system according to claim 1, wherein said user borne system includes an e-commerce payment system.

12. The user borne system according to claim 1, further comprising a spherical field-of-view camera sensor and camera sensor supporting structure configured to automatically extend in front of said user when a phone call is initiated for face-to-face video teleconferencing and retract when the phone call is completed.

13. The user borne system according to claim 1, further comprising a video teleconferencing system configured to overlay video representations of teleconference users over geographical information or imagery at each of said teleconference user'"'"'s geographical or spatial location and allow said teleconference users to interact with said geographical information or imagery.

14. The user borne system according to claim 1, wherein said user borne system is configured to operate on real-time and near real-time data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem to determine a threat to said user or said surrounding environment.

15. The user borne system according to claim 1, wherein said user borne system includes a cognitive memory system comprising a neuromorphic computing system including at least one of an analog and/or digital circuit, Application Specific Integrated Circuit (ASIC), microprocessor, or other logic in hardware, software, or firmware with auto associative artificial neural networks that at least receive and process some portion of said data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem to organize, remember, and update said data communicated to at least one of said user borne system and recipient biological, mechanical, or bio-mechanical systems such that said user borne system and/or recipient biological, mechanical, or bio-mechanical system learn through experience.

16. The user borne system according to claim 1, wherein said user mobile electronic device is a head mounted device housing said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem.

17. The user borne system according to claim 1, wherein said user borne system includes at least one memristor operable to reflect said user as a cognitive model realized as a modern dynamic system with behavioral dynamics coded into a neural network system that achieves embodied cognition in at least one of said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem, said data from said at least one of brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and/or correlation subsystem informing the user'"'"'s behavior and the user'"'"'s actions through back propagation of said neural network in an iterative repetitive manner such that said user affects said surrounding environment and said surrounding environment affects said user in perception-action cycles.

18. The user borne system according to claim 1, wherein said user borne system includes an artificial neural network configured to process said data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem.

19. The user borne system according to claim 1, wherein said user borne system includes an auto-associative neural network.

20. The user borne system according to claim 1, wherein said user borne system includes a neuromorphic system configured to process said data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem.

21. The user borne system according to claim 1, wherein said user borne system includes a self-learning neural network.

22. The user borne system according to claim 1, wherein said user borne system further comprises a portable magnetoencephalography (MEG) brain activity system configured to derive some of said data from said brain activity sensing subsystem.

23. The user borne system according to claim 1, wherein said user borne system is configured to perform artificial neural network backward propagation algorithms and processing for achieving iterative learning as said user borne system receives new data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, correlation subsystem, or recipient biological mechanical, or bio-mechanical system.

24. The user borne system according to claim 1, wherein said user borne system includes an auto-association neural network configured to perform repetitive, iterative, supervised, and unsupervised machine learning to yield an output action potential into said user borne system and/or recipient biological, mechanical, or bio-mechanical system.

25. The user borne system according to claim 1, wherein said user borne system includes a cognitive model realized as a dynamic system with behavioral dynamics coded into a neural network that achieves embodied cognition in at least one of said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, correlation subsystem, wherein said cognition embodies the user'"'"'s behavior and actions through back propagation of the neural network in an iterative and repetitive manner such that said user affects said surrounding environment and said surrounding environment affects said user in perception action cycles.

26. The user borne system according to claim 1, wherein said brain activity data is operated upon when a thought generated by said user is translated by said user borne system into at least one of text, audio, imagery, or machine language that is communicated wirelessly to said user or recipient biological, mechanical or bio-mechanical system.

27. The user borne system according to claim 1, wherein said user borne system or said recipient is configured to operate on data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, correlation subsystem, or user mobile electronic device to operate at least one actuator to assist said user or recipient system in responding to an event.

28. A system according to claim 1, wherein at least some portion of said data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem is communicated to and operated upon by a biological, mechanical, or bio-mechanical system that performs diagnostic medicine or life support.

29. A system according to claim 1, wherein said user portable system or said recipient system includes an integrated wireless communication system for command and control of said user or said recipient system from a remote location.

30. The system according to claim 1, wherein said user portable system includes a graphical user interface software or firmware application that depicts a body from which said user may interactively select which of said user brain activity sensing subsystem, user periphery sensing subsystem, and surrounding environment sensing subsystems and associated sensors said user wants to control or turn on and off.

31. A user borne system comprising:

a robotic system;

a computer system for operating said robotic system, said computer system including a neural network, said robotic system being configured to train at least a portion of said neural network by;

correlating common relationships between first sensor signatures of time, location, subject matter, and activity that define a subject in an environment based on real-time and historical data processing;

comparing second sensor signatures of time, location, subject matter, and activity that define said subject in said environment, said second sensor signatures including user queries and commands, to said first sensor signatures;

updating a relational database if significant relationships are found between said first sensor signatures and said second sensor signatures; and

automatically taking an action or notifying a user if a response is required based on a rule previously established by said user such that said computer system uses output from said neural network to learn, negotiate, and survive in said environment; and

a biological or bio-mechanical life-logging database installed on said computer system and operated upon on a non-transitory computer readable medium, said database being logged by sensors borne by said user to record perceptions of said user and surrounding environment perceptions.

32. A biological or bio-mechanical system user borne system comprising:

a brain activity sensing subsystem configured to collect data corresponding to brain activity of a biological or bio-mechanical system user;

a measurement computer subsystem configured to generate data representing quantifications of perceptions of said biological or bio-mechanical system user;

a user sensing subsystem configured to collect data corresponding to biological or bio-mechanical system user events;

a surrounding environment sensing subsystem configured to collect data corresponding to the biological or bio-mechanical system user'"'"'s surrounding environment;

a recording subsystem configured to record said data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, and surrounding environment sensing subsystem;

a correlation subsystem configured to create correlation subsystem data comprising relationships between said data corresponding to said brain activity of said user and said data corresponding to said user events and surrounding environment;

a user mobile electronic device in communication with said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, and recording subsystem, said user mobile electronic device including an interactive graphic user interface and being configured to;

operate as a host computer processing subsystem for command, control, and processing of signals to and from said brain activity sensing subsystem, user sensing subsystem, surrounding environment sensing subsystem, and correlation subsystem;

command said brain activity sensing subsystem to transmit brain activity and pattern data to said correlation subsystem; and

command said user sensing subsystem and surrounding environment sensing subsystem to transmit processed sensor data to said correlation subsystem,

said correlation subsystem being configured to receive and perform correlation processing operations to determine an extent of neural relationships between data received from said user mobile electronic device and said brain activity sensing subsystem, user sensing subsystem, and surrounding environment sensing subsystem to derive neural correlates of consciousness of conscious precepts of said user;

a non-transitory computer readable medium configured to store data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem for performing queries on real-time and near real-time data received from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem for determining whether to keep or disregard said data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem based on pre-established rule-sets and biological or bio-mechanical system user interactive command and control from said user mobile electronic device; and

a computer processing device configured to process and communicate at least a portion of said data from said brain activity sensing subsystem, measurement computer subsystem, user sensing subsystem, surrounding environment sensing subsystem, recording subsystem, and correlation subsystem into said biological or bio-mechanical system user.