Systems and Methods for Using Mobile and Wearable Video Capture and Feedback Plat-Forms for Therapy of Mental Disorders

• US 20170319123A1
• Filed: 05/08/2017
• Published: 11/09/2017
• Est. Priority Date: 05/06/2016
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Catalin Voss, Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Voss, Catalin, Haber, Nicholas Joseph, Wall, Dennis Paul, Kline, Aaron Scott, Winograd, Terry Allen
• Granted Patent: US 10,835,167 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 5/0002 Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network A61B5/07 takes precedence; transmission systems for measured values G08C; transmission H04B; transmission of digital information H04L; wireless communication networks H04W
• CPC Class Codes: A61B 5/0036 including treatment, e.g., using an implantable medical device, ablating, ventilating
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/1126 using a particular sensing technique
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/1176 Recognition of faces
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/4836 Diagnosis combined with treatment in closed-loop systems or methods A61B5/0036 takes precedence
• CPC Class Codes: A61B 5/6803 Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A61B 5/681 Wristwatch-type devices
• CPC Class Codes: A61B 5/7405 using sound
• CPC Class Codes: A61B 5/742 using visual displays A61B5/339 takes precedence
• CPC Class Codes: A61B 5/7455 characterised by tactile indication, e.g. vibration or electrical stimulation
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/945 User interactive design; Environments; Toolboxes
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G16H 20/70 : relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/63 : for local operation
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 50/30 : for calculating health indices; for individual health risk assessment

Abstract

Behavioral and mental health therapy systems in accordance with several embodiments of the invention include a wearable camera and/or a variety of sensors (accelerometer, microphone, among various other) connected to a computing system including a display, audio output, holographic output, and/or vibrotactile output to automatically recognize social cues from images captured by at least one camera and provide this information to the wearer via one or more outputs such as (but not limited to) displaying an image, displaying a holographic overlay, generating an audible signal, and/or generating a vibration.

Claims

1. An image processing system, comprising:

at least one camera for capturing images of a surrounding environment;

at least one processor and memory containing software;

wherein the software directs the at least one processor to;

obtain data comprising a sequence of images captured by the at least one camera;

detect a face for at least one person within a plurality of images in the sequence of images;

perform neutral feature estimation and subtraction on the detected face of the at least one person in each of the plurality of images and using a classifier to detect at least one emotional cue in the face based upon the plurality of images;

identify at least one emotion based on the emotional cue; and

display at least one emotion indicator label in real time to provide therapeutic feedback.

2. The image processing system of claim 1, wherein the system comprises a wearable video capture system comprising at least one outward facing camera.

3. The image processing system of claim 2, wherein the wearable video capture system is selected from the group consisting of a virtual reality headset, a mixed-reality headset, an augmented reality headset, and glasses comprising a heads-up display.

4. The image processing system of claim 2, wherein the wearable video capture system communicates with at least one mobile device, wherein the at least one processor is executing on the at least one mobile device.

5. The image processing system of claim 1, wherein the software directs the at least one processor to obtain supplementary data comprising data captured from at least one sensor selected from the group consisting of a microphone, an accelerometer, a gyroscope, an eye tracking sensor, a head-tracking sensor, a body temperature sensor, a heart rate sensor, a blood pressure sensor, and a skin conductivity sensor.

6. The image processing system of claim 1, wherein displaying at least one emotion indicator label in real time to provide therapeutic feedback further comprises performing at least one of displaying a label within a heads-up display, generating an audible signal, generating a vibration, displaying a holographic overlay, and displaying an image.

7. The image processing system of claim 1, wherein the software directs the at least one processor to process image data at a higher resolution within a region of interest related to a detected face within an image.

8. The image processing system of claim 7, wherein the region of interest is a bounding region around the detected face, wherein processing the image data further comprises using a moving average filter to smoothen the bounding region of interest.

9. The image processing system of claim 8, wherein the software directs the at least one processor to perform neutral feature estimation and subtraction on the at least one person by:

performing face detection;

performing fiducial point face tracking;

performing registration of the face to a canonical size and pose;

performing lighting normalization preprocessing;

generating a Histogram of Gradients feature extraction in a region of interest about the face;

and performing classification of the face'"'"'s expression via a logistic regression classifier; and

filtering of the resulting classification

10. The image processing system of claim 1, wherein the expression cue comprises information selected from the group consisting of facial expressions, facial muscle movements, body language, gestures, body pose, eye contact events, head pose, features of a conversation, fidgeting, and anxiety information.

11. The image processing system of claim 1, wherein the classifier is trained using a training data set of statistically representative social expression data and that provides event-based social cues.

12. The image processing system of claim 11, wherein the software directs the at least one processor to further supplement the training set with user-labeled data of target individuals.

13. The image processing system of claim 12, wherein the software directs the at least one processor to train the classifier using the training data and the user-labeled data of target individuals so that the classifier achieves higher performance detecting at least one emotional cue in the face of the target individuals compared with the performance of the classifier detecting at least one emotional cue in the face of other individuals in the training data.

14. The image processing system of claim 13, wherein the software directs the at least one processor to:

prompt a user to label data for a target individual with at least one emotional cue label; and

store the user-labeled data for the target individual in memory.

15. The image processing system of claim 1, wherein the software directs the at least one processor to store social interaction data and provide a user interface for review of the social interaction data.

16. The image processing system of claim 1, wherein the classifier is a regression machine that provides continuous social cues.

17. The image processing system of claim 1, wherein the classifier is trained as visual time-dependent classifiers using video data of standard facial expressions and with expressive talking sequences.

18. The image processing system of claim 1, wherein the software directs the at least one processor to detect gaze events using at least one inward-facing eye tracking data in conjunction with outward-facing video data.

19. The image processing system of claim 1, wherein the software directs the at least one processor to provide a review of activities recorded and provide user behavioral data generated as a reaction to the recorded activities.

20. The image processing system of claim 1, wherein performing neutral feature estimation and subtraction on the face of the at least one person comprises performing neutral expression subtraction based upon outputs of a first expression classifier trained on neutral-subtracted features and a second 2-class neutral/expressive classifier trained on raw (not neutral-subtracted) features.

21. A behavioral therapy system, comprising:

at least one camera for capturing images of a surrounding environment;

at least one processor and memory containing software;

wherein the software directs the at least one processor to

display, for each of a plurality of images previously stored within the memory of the system, an image of a face of person expressing a particular emotion, wherein the image is associated with the particular emotion;

receive an input from a user viewing the image regarding an emotion that the user has selected as illustrating the emotion being portrayed by the face of the person;

determine whether the received input from the user matches the particular emotion associated with the particular image; and

provide feedback to the user based on their selections.

22. A behavioral therapy system, comprising:

at least one camera for capturing images of a surrounding environment;

at least one processor and memory containing software;

wherein the software directs the at least one processor to;

capture video of a person in real-time;

detect an emotion for a face of the person using at least one classifier;

provide an indication regarding a different emotion that should be provoked in the person that is currently not being detected;

determine whether the emotion has been elicited within a certain time period; and

provide feedback to a user of the system regarding their ability to elicit the emotion.