Health monitoring system and appliance

• US 10,235,998 B1
• Filed: 02/28/2018
• Issued: 03/19/2019
• Est. Priority Date: 02/28/2018
• Status: Active Grant

• Current Assignee: The Notebook, LLC
• Original Assignee: Karen Elaine Khaleghi
• Inventors: Khaleghi, Karen Elaine
• Primary Examiner(s): Sirjani, Fariba
• Application Number:
  US15/908,428
• Time in Patent Office: 384 Days
• Field of Search: None
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/4803 Speech analysis specially adapted for diagnostic purposes
• CPC Class Codes: B60K 28/06 responsive to incapacity of driver
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/26 Incapacity
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/24 the extracted parameters being the cepstrum
• CPC Class Codes: G10L 25/66 : for extracting parameters related to health condition detecting or measuring for diagnostic purposes A61B5/00
• CPC Class Codes: G10L 25/78 : Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: G10L 25/90 : Pitch determination of speech signals
• CPC Class Codes: G10L 25/93 : Discriminating between voiced and unvoiced parts of speech signals G10L25/90 takes precedence

Abstract

Systems and methods are disclosed. A digitized human vocal expression of a user and digital images are received over a network from a remote device. The digitized human vocal expression is processed to determine characteristics of the human vocal expression, including: pitch, volume, rapidity, a magnitude spectrum identify, and/or pauses in speech. Digital images are received and processed to detect characteristics of the user face, including detecting if one or more of the following is present: a sagging lip, a crooked smile, uneven eyebrows, and/or facial droop. Based at least on part on the human vocal expression characteristics and face characteristics, a determination is made as to what action is to be taken. A cepstrum pitch may be determined using an inverse Fourier transform of a logarithm of a spectrum of a human vocal expression signal. The volume may be determined using peak heights in a power spectrum of the human vocal expression.

Claims

1. An electronic device configured to process audible expressions from users, comprising:

a network interface;

a haptic engine configured to provide kinesthetic communication;

at least one computing device; and

computer readable memory including instructions operable to be executed by the at least one computing device to perform a set of actions, configuring the at least one computing device to;

receive in real time, over a network via the network interface, a digitized human vocal expression of a first user and one or more digital images from a remote device;

process the received digitized human vocal expression using digital signal processing to convert the digitized audible expression from a time domain to a frequency domain, and to perform at least one of dimensionality reduction or warping of two or more frequencies to a first scale thereby reducing an amount of vocal expression data that needs to be processed;

use the processed digitized human vocal expression to determine characteristics of the human vocal expression, including;

determine, using a pitch analysis module, a pitch of the human vocal expression,

determine, using a volume analysis module a volume of the human vocal expression,

determine, using a rapidity analysis module how rapidly the first user is speaking in the human vocal expression,

determine, using a vocal tract analysis module, a magnitude spectrum of the human vocal expression, and

identify, using a non-speech analysis module, pauses and the length of pauses in speech in the human vocal expression;

use a natural language module to convert audible speech in the human vocal expression to text and to understand audible speech in the human vocal expression;

compare the determined characteristics of the human vocal expression with baseline, historical characteristics of human vocal expressions associated with the first user to identify changes in human vocal expression characteristics of the first user;

process the received one or more images to detect characteristics of the first user face, including detecting if one or more of the following are present;

a sagging lip, a crooked smile, uneven eyebrows, facial droop;

compare the detected characteristics of the first user face with baseline, historical characteristics of the first user face accessed from a data store, and identify changes in characteristics of the first user face;

weight, using a first weight, a first identified change with respect to a first vocal expression characteristic of the first user;

weight, using a second weight, a second identified change with respect to a second vocal expression characteristic of the first user;

weight, using a third weight, a third identified change with respect to a first characteristic of the first user face;

weight, using a fourth weight, a fourth identified change with respect to a second characteristic of the first user face;

inferring a change in health status of the first user based at least in part on the weighted first identified change with respect to the first vocal expression characteristic of the first user, the weighted second identified change with respect to the second vocal expression characteristic of the first user, the weighted third identified change with respect to the first characteristic of the first user face, the weighted fourth identified change with respect to the second characteristic of the first user face;

based at least in part on the inferred change in health status of the first user determine if a vehicle is to be deployed to the first user; and

at least partly in response to a determination that a vehicle is to be deployed to the first user, enable a vehicle to be deployed to a location of the first user.

2. The electronic device as defined in claim 1, wherein the pitch analysis module is configured to estimate a quasiperiodic signal period of the human vocal expression and determine the pitch using the estimated quasiperiodic signal period.

3. The electronic device as defined in claim 1, wherein the pitch analysis module is configured to determine a cepstrum pitch using an inverse Fourier transform (IFT) of a logarithm of an estimated spectrum of a human vocal expression signal.

4. The electronic device as defined in claim 1, wherein the volume analysis module is configured to determine the volume of the human vocal expression based at least in part on peak heights in a power spectrum of the human vocal expression.

5. The electronic device as defined in claim 1, wherein the rapidity analysis module is configured to determine how rapidly the first user is speaking based at least in part on a determination of how many words are spoken by the first user over a first period of time.

6. The electronic device as defined in claim 1, wherein the non-speech analysis module is configured to identify pauses in speech in the human vocal expression using a power and/or magnitude spectrum of the human vocal expression.

7. The electronic device as defined in claim 1, wherein the at least one computing device is configured to determine if an occlusion of eyes of the first user by eyelids of the first user indicates an adverse health state.

8. An electronic device, comprising:

a network interface;

a haptic engine configured to provide kinesthetic communication;

at least one computing device; and

computer readable memory including instructions operable to be executed by the at least one computing device to perform a set of actions, configuring the at least one computing device to;

receive, over a network via the network interface, a digitized human vocal expression of a first user;

convert at least a portion of the digitized human vocal expression to text;

process the received digitized human vocal expression using digital signal processing to convert the digitized audible expression from a time domain to a frequency domain, and to perform at least one of dimensionality reduction or warping of two or more frequencies to a first scale;

use the processed digitized human vocal expression to determine characteristics of the human vocal expression, including;

determine a pitch of the human vocal expression,

determine a volume of the human vocal expression,

determine how rapidly the first user is speaking in the human vocal expression,

determine a magnitude and/or power spectrum of the human vocal expression,

determine pauses and the length of pauses in speech in the human vocal expression, and

analyze lexicon usage, syntax, semantics, and/or discourse patterns in speech in the human vocal expression;

compare the determined characteristics of the human vocal expression with baseline, historical characteristics of human vocal expressions associated with the first user to identify changes in human vocal expression characteristics of the first user;

weight, using a first weight, a first identified change with respect to a first vocal expression characteristic of the first user;

weight, using a second weight, a second identified change with respect to a second vocal expression characteristic of the first user;

inferring a change in health status of the first user based at least in part on the weighted first identified change with respect to the first vocal expression characteristic of the first user, the weighted second identified change with respect to the second vocal expression characteristic of the first user;

based at least in part on the inferred change in health status of the first user,

determine if a first action is to be taken.

9. The electronic device as defined in claim 8, wherein the at least one computing device is configured to:

receive one or more digital images;

process the received one or more images to detect characteristics of the first user face, including detecting if one or more of the following is present;

a sagging lip, a crooked smile, uneven eyebrows, facial droop;

wherein the determination as to whether the first action is to be taken is based in part on one or more detected characteristics of the first user face.

10. The electronic device as defined in claim 8, wherein the electronic device comprises a vehicle, and the first action comprises causing the vehicle to be prevented from being drivable or flyable.

11. The electronic device as defined in claim 8, wherein the electronic device is configured to estimate a quasiperiodic signal period of the human vocal expression and determine the pitch using the estimated quasiperiodic signal period.

12. The electronic device as defined in claim 8, wherein the electronic device is configured to determine a cepstrum pitch using an inverse Fourier transform (IFT) of a logarithm of an estimated spectrum of a human vocal expression signal.

13. The electronic device as defined in claim 8, wherein the electronic device is configured to determine the volume of the human vocal expression based at least in part on peak heights in a power spectrum of the human vocal expression.

14. The electronic device as defined in claim 8, wherein the electronic device is configured to determine how rapidly the first user is speaking based at least in part on a determination of how many words are spoken over a first period of time.

15. The electronic device as defined in claim 8, wherein the electronic device is configured to identify pauses in speech in the human vocal expression using a power and/or magnitude spectrum of the human vocal expression.

16. The electronic device as defined in claim 8, wherein the electronic device is configured to determine if an occlusion of eyes of the first user by eyelids of the first user indicates an adverse health state.

17. The electronic device as defined in claim 8, wherein the electronic device is configured to:

receive one or more digital images of the first user;

process the received one or more images to detect characteristics of the first user face, including detecting if at least one of the following is present;

a sagging lip, a crooked smile, uneven eyebrows, or facial droop;

wherein the first action comprises generation of a notification and provision of the notification to one or more destinations, wherein the notification comprises;

at least a portion of the received digitized human vocal expression,

text corresponding to at least a portion of the received digitized human vocal expression;

at least one received image; and

a facial feature analysis.

18. A computer implemented method, comprising:

communicating with at least one user using a haptic engine;

receiving, at a system configured to process digitized human vocal expressions using a digital signal processing module, a digitized human vocal expression of a first user from a first user device;

converting by the system at least a portion of the digitized human vocal expression to text;

processing, using the digital signal processing module, the received digitized human vocal expression to convert the digitized audible expression from a time domain to a frequency domain, and to perform at least one of dimensionality reduction or warping of two or more frequencies to a first scale;

using, by the system, the processed digitized human vocal expression to determine characteristics of the human vocal expression, including;

determining a pitch of the human vocal expression,

determining a volume of the human vocal expression,

determining how rapidly the first user is speaking in the human vocal expression,

determining a magnitude and/or power spectrum of the human vocal expression,

determining pauses and the length of pauses in speech in the human vocal expression, and

analyzing lexicon usage, syntax, semantics, and/or discourse patterns in speech in the human vocal expression;

comparing one or more of the determined characteristics of the human vocal expression with one or more baseline, historical characteristics of human vocal expressions associated with the first user;

weighting by the system, using a first weight, a first identified change with respect to a first vocal expression characteristic of the first user;

weighting by the system, using a second weight, a second identified change with respect to a second vocal expression characteristic of the first user;

inferring, by the system, a change in health status of the first user based at least in part on the weighted first identified change with respect to the first vocal expression characteristic of the first user, the weighted second identified change with respect to the second vocal expression characteristic of the first user;

based at least in part on the inferred change in health status of the first user, determining if a first action is to be taken.

19. The method as defined in claim 18, the method further comprising:

receiving one or more digital images from the first user device;

processing the received one or more images to detect characteristics of the first user face, including detecting if one or more of the following is present;

a sagging lip, a crooked smile, uneven eyebrows, facial droop;

wherein the determination as to whether the first action is to be taken is based in part on the detected characteristics of the first user face.

20. The method as defined in claim 18, wherein the first user device comprises a vehicle, and the first action comprises causing the vehicle to be prevented from being drivable or flyable.

21. The method as defined in claim 18, the method further comprising estimating a quasiperiodic signal period of the human vocal expression and determine the pitch using the estimated quasiperiodic signal period.

22. The method as defined in claim 18, the method further comprising determining the volume of the human vocal expression based at least in part on peak heights in a power spectrum of the human vocal expression.

23. The method as defined in claim 18, the method further comprising identifying pauses in speech in the human vocal expression using a power spectrum of the human vocal expression.

24. The method as defined in claim 18, the method further comprising:

receiving one or more digital images from the first user device;

processing the received one or more images to detect occlusion of eyes of the first user by eyelids of the first user;

determining whether an occlusion of eyes of the first user by eyelids of the first user indicates an adverse health state;

wherein the determination as to whether the first action is to be taken is based in part on the determination of whether an occlusion of eyes of the first user by eyelids of the first user indicates an adverse health state.

25. The method as defined in claim 18, the method further comprising:

receiving one or more digital images from the first user device;

processing the received one or more images to detect characteristics of the first user face, including detecting if at least one of the following is present;

a sagging lip, a crooked smile, uneven eyebrows, or facial droop;

wherein the first action comprises generating a notification and providing the notification to one or more destinations, wherein the notification comprises;

at least a portion of the received digitized human vocal expression,

text corresponding to at least a portion of the received digitized human vocal expression;

at least one received image; and

a facial feature analysis.

26. A computer implemented method, comprising:

communicating with at least one user using a haptic engine;

receiving, at a computerized device configured to process digitized human vocal expressions using a digital signal processing module, a digitized human vocal expression of a first user;

processing, using the digital signal processing module, the received digitized human vocal expression to convert the digitized audible expression from a time domain to a frequency domain, and to perform at least one of dimensionality reduction or warping of two or more frequencies to a first scale;

using, by the system, the processed digitized human vocal expression to determine characteristics of the human vocal expression, including;

determining a volume of the human vocal expression,

determining how rapidly the first user is speaking in the human vocal expression,

generating spectrum analysis of the human vocal expression,

determining pauses and the length of pauses in speech in the human vocal expression, and

analyzing lexicon usage, syntax, semantics, and/or discourse patterns in speech in the human vocal expression;

comparing, using the computerized device, one or more of the determined characteristics of the human vocal expression with one or more baseline, historical characteristics of human vocal expressions associated with the first user;

weighting by the system, using a first weight, a first identified change with respect to a first vocal expression characteristic of the first user;

weighting by the system, using a second weight, a second identified change with respect to a second vocal expression characteristic of the first user;

inferring, by the system, a change in health status of the first user based at least in part on the weighted first identified change with respect to the first vocal expression characteristic of the first user, the weighted second identified change with respect to the second vocal expression characteristic of the first user;

based at least in part on the inferred change in health status of the first user,

determining if a first action is to be taken.

27. The method as defined in claim 26, the method further comprising:

receiving one or more digital images;

processing the received one or more images to detect characteristics of the first user face, including detecting if one or more of the following is present;

a sagging lip, a crooked smile, uneven eyebrows, facial droop;

wherein the determination as to whether the first action is to be taken is based in part on the detected characteristics of the first user face.

28. The method as defined in claim 26, wherein the computerized device comprises a vehicle, and the first action comprises causing the vehicle to be prevented from being drivable or flyable.

29. The method as defined in claim 26, the method further comprising determine how rapidly the first user is speaking based at least in part on a determination of how many words are spoken over a first period of time.

30. The method as defined in claim 26, the first action comprising generating a notification and providing the notification to one or more destinations, the notification comprising:

text corresponding to at least a portion of the received digitized human vocal expression, the text generated from the received digitized human vocal expression utilizing a speech-to-text module,

at least a portion of the received digitized human vocal expression, and

received video content of the first user, comprising one or more digital images of the first user.