WEARABLE COMPUTING APPARATUS AND METHOD

• US 20140347265A1
• Filed: 03/17/2014
• Published: 11/27/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: InteraXon Inc.
• Original Assignee: InteraXon Inc.
• Inventors: AIMONE, CHRISTOPHER ALLEN, GARTEN, ARIEL STEPHANIE, COLEMAN, TREVOR, PINO, LOCILLO (LOU) GIUSEPPE, VIDYARTHI, KAPIL JAY MISHRA, BARANOWSKI, PAUL HARRISON, CHABIOR, MICHAEL APOLLO, CHONG, TRACY, RUPSINGH, RAUL RAJIV, ASHBY, MADELINE, TADICH, PAUL V.
• Granted Patent: US 10,365,716 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A61B 2560/0493 controlled by voice
• CPC Class Codes: A61B 5/0024 for multiple sensor units attached to the patient, e.g. using a body or personal area network
• CPC Class Codes: A61B 5/02416 using photoplethysmograph signals, e.g. generated by infrared radiation A61B5/14552 takes precedence
• CPC Class Codes: A61B 5/02438 with portable devices, e.g. worn by the patient
• CPC Class Codes: A61B 5/0261 using optical means, e.g. infrared light
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/375 using biofeedback
• CPC Class Codes: A61B 5/378 Visual stimuli
• CPC Class Codes: A61B 5/38 Acoustic or auditory stimuli
• CPC Class Codes: A61B 5/389 Electromyography [EMG]
• CPC Class Codes: A61B 5/398 Electrooculography [EOG], e.g. detecting nystagmus; Electroretinography [ERG]
• CPC Class Codes: A61B 5/4064 Evaluating the brain A61B5/031, A61B5/369, A61B5/14553 take precedence
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A61B 5/744 Displaying an avatar, e.g. an animated cartoon character
• CPC Class Codes: A61M 2021/0022 by the tactile sense, e.g. vibrations
• CPC Class Codes: A61M 2021/0027 by the hearing sense
• CPC Class Codes: A61M 2021/0044 by the sight sense
• CPC Class Codes: A61M 2021/0066 with heating or cooling
• CPC Class Codes: A61M 21/00 : Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis
• CPC Class Codes: A61M 2230/10 : Electroencephalographic signals
• CPC Class Codes: A61M 2230/14 : Electro-oculogram [EOG]
• CPC Class Codes: G02B 2027/0187 : slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 : Head mounted
• CPC Class Codes: G02C 11/10 : Electronic devices other than hearing aids
• CPC Class Codes: G02C 7/027 : considering wearer's parameters
• CPC Class Codes: G06F 16/90 : Details of database functions independent of the retrieved data types
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 : Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/048 : Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0487 : using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G09G 3/003 : to produce spatial visual effects
• CPC Class Codes: G16H 20/70 : relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/67 : for remote operation
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04M 1/05 : specially adapted for use on head, throat or breast
• CPC Class Codes: H04M 2250/12 : including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services
• CPC Class Codes: H04W 4/30 : Services specially adapted for particular environments, situations or purposes

Abstract

A method is provided, performed by a wearable computing device comprising at least one bio-signal measuring sensor, the at least one bio-signal measuring sensor including at least one brainwave sensor, comprising: acquiring at least one bio-signal measurement from a user using the at least one bio-signal measuring sensor, the at least one bio-signal measurement comprising at least one brainwave state measurement; processing the at least one bio-signal measurement, including at least the at least one brainwave state measurement, in accordance with a profile associated with the user; determining a correspondence between the processed at least one bio-signal measurement and at least one predefined device control action; and in accordance with the correspondence determination, controlling operation of at least one component of the wearable computing device, such as modifying content displayed on a display of the wearable computing device. Various types of bio-signals, including brainwaves, may be measured and used to control the device in various ways.

Claims

1. A method, performed by a wearable computing device comprising at least one bio-signal measuring sensor, the at least one bio-signal measuring sensor including at least one brainwave sensor, comprising:

acquiring at least one bio-signal measurement from a user using the at least one bio-signal measuring sensor, the at least one bio-signal measurement comprising at least one brainwave state measurement;

processing the at least one bio-signal measurement, including at least the at least one brainwave state measurement, in accordance with a profile associated with the user;

determining a correspondence between the processed at least one bio-signal measurement and at least one predefined device control action; and

in accordance with the correspondence determination, controlling operation of at least one component of the wearable computing device.

2. The method of claim 1 wherein the at least one component comprises a display, the controlling operation of the at least one component comprises modifying or initiating the modification of an image displayed on the display.

3. The method of claim 2 wherein the wearable computing device comprises at least one eye-tracking sensor, the method comprising:

displaying at least one item on the display;

acquiring at least one eye-tracking measurement from the user using the at least one eye-tracking sensor;

determining a correspondence between the eye-tracking measurement and the displayed location of one of the at least one displayed item; and

in accordance with the eye-tracking correspondence determination and the bio-signal correspondence determination, modifying or initiating the modification of at least one displayed item by the at least one predefined device control action.

4. The method of claim 3 wherein the at least one item comprises a search result item, and the at least one predefined device control action comprises selecting the one of the at least one displayed item;

wherein the at least one predefined control action comprises narrowing the search results by at least one property associated with the one at least one displayed item.

5. The method of claim 3 wherein the processed at least one bio-signal measurement identifies a user-interest in the one at least one displayed item by comparing the bio-signal measurement to a predefined bio-signal measurement stored in the user profile, the bio-signal correspondence determination based at least partly on the identified user-interest.

6. The method of claim 3 wherein the bio-signal correspondence determination determines a correspondence between the at least one bio-signal measurement and a predefined bio-signal measurement stored in the user profile, the predefined bio-signal measurement associated with at least one emotional response type.

7. The method of claim 6 wherein the at least one predefined display control action comprises tagging the one at least one displayed item with the corresponding at least one emotional response type.

8. The method of claim 2 comprising displaying at least one item on the display, each of the at least one displayed item displayed at a distinct display frequency with respect to the other at least one displayed item, wherein the correspondence determining comprises determining a correspondence between the at least one brainwave state measurement and the display frequency of the at least one displayed item.

9. The method of claim 2 comprising displaying at least one item on the display, the at least one item associated with an item sequence;

wherein the processed at least one bio-signal measurement identifies a user-interest in the one at least one displayed item by comparing the bio-signal measurement to a predefined bio-signal measurement stored in the user profile associated with a predetermined user-interest level, and wherein the correspondence determining comprises determining a correspondence between the processed at least one bio-signal measurement and the at least one predefined device control action based at least partly on the identified user interest.

10. The method of claim 9 wherein the at least one predefined device control action comprises modifying a rate of advancement in displaying a subsequent at least one display item in the item sequence in accordance with a determined level of the identified user-interest.

11. The method of claim 1 wherein the computing device comprises a human-wearable eyeglass frame, each of the display and bio-signal measuring sensor being connected to the eyeglass frame.

12. The method of claim 1 wherein the wearable computing device comprises at least one camera, the method comprising:

in accordance with a determined correspondence between the processed at least one bio-signal measurement and a predetermined bio-signal state, controlling the at least one camera to take visual input of a current field of view of the user of the wearable computing device.

13. The method of claim 12 comprising receiving input from the user indicating at least one object viewable in the field of view;

and associating the at least one bio-signal measurement with the indicated at least one object;

wherein the controlling operation comprises modifying or initiating the modification of an image displayed on the display based at least partly on the indicated at least one object.

14. The method of claim 13 wherein the wearable computing device comprises at least one eye-tracking sensor, the method comprising acquiring at least one eye-tracking measurement from the user using the at least one eye-tracking sensor;

and determining the indicated at least one object at least partly by determining at least one eye focal area in the field of view based at least partly on the at least one eye-tracking measurement.

15. The method of claim 12 comprising identifying at least one object in the field of view at least partly by applying at least one image recognition process to the visual input from the camera;

and the controlling operation comprising modifying or initiating the modification of an image displayed on the display is based at least partly on the identified at least one object.

16. The method of claim 15 wherein the modifying or initiating the modification of an image displayed on the display comprises displaying information associated with the identified object retrieved from a database.

17. The method of claim 12 wherein the wearable computing device comprises:

at least one location-determining component; and

at least one orientation-determining component;

the method comprising;

determining a field of view of a wearer of the wearable computing device based at least partly on;

a determined location of the wearable computing device; and

a determined orientation of the wearable computing device, the determined location and the determined orientation based at least partly on data received from the at least one location-determining component and the at least one orientation-determining component; and

determining at least one object that is viewable based at least partly on the determined field of view and a database of objects associated with at least one location determined to be viewable in the determined field of view;

wherein the modifying or initiating the modification of an image displayed on the display comprises displaying information associated with the identified object retrieved from a database.

18. The method of claim 12 comprising identifying at least one object in the field of view at least partly by applying at least one image recognition process to the visual input from the camera;

wherein the controlling operation comprising modifying or initiating the modification of an image displayed on the display is based at least partly on the identified at least one object.

19. The method of claim 12 comprising associating the visual input with the at least one brainwave state measurement and updating the profile with the associated visual input.

20. The method of claim 1 wherein the controlling operation of at least one component of the wearable computing device comprises sharing the processed at least one brainwave state measurement with at least one computing device over a communications network.

21. The method of claim 20 wherein the wearable computing device comprises at least one camera;

wherein controlling operation of at least one component of the wearable computing device comprises sharing visual input received from the at least one camera with the at least one computing device;

the method comprising receiving directions from the at least one computing device in response to the shared visual input and at least one brainwave state measurement.

22. The method of claim 2 wherein the wearable computing device comprises a camera, the method comprising:

identifying at least one person viewable in visual input from the camera by applying at least one image recognition process to the visual input; and

modifying or initiating the modification of the image displayed on the display with information associated with the identified at least one person.

23. The method of claim 22 comprising receiving the information associated with the identified at least one person from at least one computer server, the information comprising at least the identified person'"'"'s name.

24. The method of claim 22 comprising determining at least one emotional state of the at least one person viewable in the visual input from the camera based at least partly on at least one voice stress analysis, facial expression, body language, and changes in skin colour as analyzed by the wearable computing device.

25. The method of claim 22 comprising receiving at least one brainwave state measurement of the at least one person from at least one second wearable computing device over a communications network;

and modifying or initiating the modification of the image displayed on the display based at least partly on the received at least one brainwave state measurement of the at least one person.

26. The method of claim 2 wherein the processed at least one brainwave state measurement is indicative of error-related negativity and the at least one predefined device control action comprises displaying information on the display related to modify application of a previously applied predefined device control action.

27. A non-transitory computer program product tangibly embodying code that, when executed by a processor, causes the processor to:

acquire at least one bio-signal measurement from a user using at least one bio-signal measuring sensor, the at least one bio-signal measurement comprising at least one brainwave state measurement;

process the at least one bio-signal measurement, including at least the at least one brainwave state measurement, in accordance with a profile associated with the user;

determine a correspondence between the processed at least one bio-signal measurement and at least one predefined device control action; and

in accordance with the correspondence determination, control operation of at least one component of a wearable computing device.

28. A wearable computing device comprising:

at least one bio-signal measuring sensor, the at least one bio-signal measuring sensor including at least one brainwave sensor;

at least one processor coupled to the at least one bio-signal measuring sensor; and

a non-transitory computer-readable medium or media comprising computer-executable instructions configured to cause the at least one data processor to;

acquire at least one bio-signal measurement from a user using the at least one bio-signal measuring sensor, the at least one bio-signal measurement comprising at least one brainwave state measurement;

process the at least one bio-signal measurement, including at least the at least one brainwave state measurement, in accordance with a profile associated with the user;

determine a correspondence between the processed at least one bio-signal measurement and at least one predefined device control action; and

in accordance with the correspondence determination, control operation of at least one component of the wearable computing device.