Voice controlled assistance for monitoring adverse events of a user and/or coordinating emergency actions such as caregiver communication

• US 10,258,295 B2
• Filed: 05/09/2017
• Issued: 04/16/2019
• Est. Priority Date: 05/09/2017
• Status: Active Grant

• Current Assignee: Charles Goldman, Peter Mitchell, Richard D. Lane, Stuart R. Patterson, Voice Care Tech Holdings LLC
• Original Assignee: Lifepod Solutions Incorporated
• Inventors: Fountaine, Dennis
• Primary Examiner(s): Washburn, Daniel C
• Assistant Examiner(s): Ogunbiyi, Oluwadamilola M
• Application Number:
  US15/590,012
• Publication Number:
  US 20180325469A1
• Time in Patent Office: 707 Days
• CPC Class Codes: A61B 2560/0242 adapted to measure environmental factors, e.g. temperature, pollution
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1117 Fall detection
• CPC Class Codes: A61B 5/747 in case of emergency, i.e. alerting emergency services
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 50/22 Social work or social welfare, e.g. community support activities or counselling services
• CPC Class Codes: G08B 17/10 Actuation by presence of smoke or gases , e.g. automatic alarm devices for analysing flowing fluid materials by the use of optical means
• CPC Class Codes: G08B 21/0407 based on behaviour analysis
• CPC Class Codes: G08B 21/0415 detecting absence of activity per se
• CPC Class Codes: G08B 21/0423 detecting deviation from an expected pattern of behaviour or schedule
• CPC Class Codes: G08B 21/043 detecting an emergency event, e.g. a fall
• CPC Class Codes: G08B 21/0446 worn on the body to detect changes of posture, e.g. a fall, inclination, acceleration, gait
• CPC Class Codes: G08B 21/0469 Presence detectors to detect unsafe condition, e.g. infrared sensor, microphone G08B21/0476 takes precedence
• CPC Class Codes: G08B 25/016 Personal emergency signalling and security systems emergency non-personal manually actuated alarm activators G08B25/12
• CPC Class Codes: G08B 7/066 guiding along a path, e.g. evacuation path lighting strip
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 25/63 : for estimating an emotional state
• CPC Class Codes: G10L 25/66 : for extracting parameters related to health condition detecting or measuring for diagnostic purposes A61B5/00
• CPC Class Codes: G16H 40/67 : for remote operation
• CPC Class Codes: H04N 21/2187 : Live feed
• CPC Class Codes: H04W 4/90 : Services for handling of emergency or hazardous situations, e.g. earthquake and tsunami warning systems [ETWS]

Abstract

Disclosed is a method, a device, a system and/or a manufacture of voice controlled assistance for monitoring adverse events of a user and/or coordinating emergency actions such as caregiver communication. In one embodiment, a system includes an assistance device coupled to a voice controlled assistance server over a network. The assistance device includes a memory including computer-executable instructions that when executed, upon an event such as sensing the presence of the user, generates a verbal status query to the user usable to determine an event of concern and/or an adverse event related to the user, such as a negative response or non-response event that may require initiation of an emergency action. The system may further include one or more sensing pods to gather a response and/or establish communication with of the user within a caregiving environment, a scheduling server to streamline interaction with the voice controlled assistance server, one or more devices (of the user, the caregiver, and/or the emergency service), and an emergency server for providing a specialized emergency assistance upon determination of the adverse event.

Claims

1. An apparatus comprising:

a speaker for generating a verbal status query to a user;

a microphone to gather from an immediate environment of the microphone at least one of a voice communication of a user and an ambient sound of the immediate environment;

a network interface controller providing communicative coupling to at least one instance of a voice controlled assistance service and at least one instance of an automated emergency assistance service;

a presence sensor;

a processor;

a memory comprising computer-executable instructions that when executed on the processor cause the processor to;

generate the verbal status query for the user in response to a sensing event of the presence sensor sensing a presence of the user and upon detection of at least one of a sound spike and a vibration spike;

determine occurrence of an event of concern comprising a non-response event defined as a failure to receive by the microphone a recognizable response from the user;

repeat the verbal status query to the user and increase a volume of the verbal status query from the speaker following determination of the non-response event of the user;

upon occurrence of the event of concern, transmit a status query to a device of the user;

generate a verbal warning to the user with the speaker that at least one of;

(i) the user has failed to provide the recognizable response, (ii) the caregiver has been alerted, and (iii) the automated emergency assistance service has been alerted;

determine that the event of concern is an adverse event requiring transmission of a notification to a device of a caregiver of the user;

initiate an alert message notification to the device of the caregiver of the user based on the adverse event; and

upon determination of the adverse event, automatically re-route a destination of a set of voice communications of the user from a first voice controlled assistance service that provides a general assistance to a second voice controlled assistance service comprising an automated emergency assistance service; and

a database to store;

a safety profile specifying a condition under which one or more events of concern define the adverse event requiring notification to the device of the caregiver of the user.

2. The apparatus of claim 1, further comprising:

a particulate sensor to detect within the environment of the user a particulate associated with a first health risk to the user,

a chemical sensor to detect within the environment of the user a chemical associated with a second health risk to the user,

the memory comprising computer-executable instructions that when executed on the processor cause the processor to;

detect at least one of the particulate at the particulate sensor and the chemical at the chemical sensor; and

determine an adverse event requiring execution of an emergency action based on the detection of at least one of the particulate at the particulate sensor and the chemical at the chemical sensor.

3. The apparatus of claim 2, further comprising:

a user interface, to display a textual message converted from a speech response of the voice controlled assistance service;

the memory further comprising a speech-text converter comprising computer-executable instructions that when executed on the processor cause the processor to;

convert the speech response of the voice controlled assistance service to the textual message for display on a graphical user interface;

a video camera,

the memory further comprising computer-executable instructions that when executed on the processor cause the processor to;

upon occurrence of the adverse event, establish a video feed between the apparatus and at least one of;

(i) the automated emergency assistance service and (ii) the device of the caregiver;

the memory further comprising computer-executable instructions that when executed on the processor cause the processor to;

populate the graphical user interface with the textual message converted with the speech-text converter for presentation to the user,

determine a threshold inactivity of the device of the user;

wherein the presence sensor is at least one of a motion sensor, an infrared sensor, a proximity sensor, an RFID sensor, a noise sensor, a Bluetooth sensor, and an ultrasonic frequency sound sensor;

wherein the chemical sensor is at least one of a carbon dioxide sensor and a carbon monoxide sensor;

wherein the particulate sensor is a smoke detector;

wherein the event of concern further comprising a negative response event of the user defined as a negative response from the user to the verbal status query;

wherein the threshold inactivity is determined based on at least one of a GPS coordinate of the device, a network traffic of the device of the user, an accelerometer data of the device of the user, a microphone of the device of the user, and a video feed of the device of the user.

4. The apparatus of claim 1,

the memory comprising an assistance service scheduler that comprises computer-executable instructions that when executed on the processor cause the processor to:

define a custom command word and associating the custom command word with an assistance application of a voice controlled assistance service along with an assistance action of the assistance application, the assistance action conforming to a voice command protocol of the assistance application;

pre-specify one or more voice inputs required upon initiation of the assistance action to execute the assistance action of the assistance application, the one or more voice inputs conforming to the voice command protocol of the assistance application;

specify at least one of (i) a confirmation requirement for a verbal confirmation of the user prior to execution of the assistance application and (ii) an input request for one or more additional verbal inputs conforming to the voice command protocol before execution of the assistance application;

schedule execution of the assistance action of the assistance application, wherein the event of concern further comprising at least one of failure to receive the verbal confirmation of the user and a failure to receive the input request for the one or more additional verbal inputs.

5. A system comprising:

an assistance device communicatively coupled to a voice controlled assistance service over a network, the assistance device comprising;

a speaker of the assistance device,

a microphone of the assistance device,

a processor of the assistance device,

a presence sensor of the assistance device, to sense a presence of a user of the assistance device within an immediate environment of the assistance device, and

a memory of the assistance device, comprising computer-executable instructions that when executed on the processor cause the processor to;

upon sensing the presence of the user, generate a verbal status query to the user comprising an authorization request for execution of an assistance action of an assistance application of the voice controlled assistance service, the assistance action conforming to a voice command protocol of the assistance application;

invoke the assistance application and initiate the assistance action upon receiving an authorization from the user;

determine an adverse event of the user based on an event of concern that is at least one of a non-response event of the user to the verbal status query generated by the speaker of the assistance device and a negative response event of the user to the verbal status query generated by the speaker of the assistance device; and

initiate notification to a device of a caregiver of the user based on the determination of the adverse event;

a sensing pod to expand communication between the user and the assistance device to an extended environment, the sensing pod comprising;

a microphone of the sensing pod to gather a sound signal from an immediate environment of the sensing pod that may be determined by the assistance device to be at least one of a non-response event of the user to the verbal status query and a negative response event of the user defined as a negative response from the user to the verbal status query;

a speaker of the sensing pod, optionally to generate the verbal status query;

a processor of the sensing pod;

a memory of the sensing pod; and

a wireless transmitter-receiver for communicative coupling with the assistance device over the network to transmit the sound signal;

a mobile device comprising;

a processor of the mobile device;

a memory of the mobile device comprising computer-executable instructions that when executed on the processor cause the processor to;

receive a status query generated upon the non-response event of the user to the verbal status query generated by the speaker;

a wearable device, comprising;

a presence signaler detectable by the presence sensor of the assistance device, and

a memory of the wearable device comprising computer-executable instructions that when executed on the processor cause the processor to;

receive the status query generated upon the non-response event of the user to the verbal status query generated by the speaker; and

an emergency server, comprising;

a processor of the emergency server;

an emergency speech recognition engine specialized in an emergency speech; and

a memory of the emergency server comprising computer-executable instructions that when executed on the processor cause the processor to;

receive a set of voice communications of the user re-routed exclusively to the emergency server upon determination of at least one of the event of concern and the adverse event,

increase a recognition sensitivity to an emergency word of the user,

generate an alert message for at least one of the emergency server and the device of the caregiver of the user, the alert message comprising a log comprising at least one of the adverse event and the event of concern, and

upon occurrence of the adverse event, establish an audio feed between the assistance device and at least one of;

(i) an emergency service and (ii) the device of the caregiver.

6. The system of claim 5, further comprising:

an automated assistance server for providing a general voice assistance of the voice controlled assistance service, the automated assistance server comprising;

a processor of the automated assistance server,

a memory of the automated assistance server,

a speech recognition engine to parse a voice communication of the user;

a personal assistance engine to provide at least one of an action in response to the voice communication of the user and a speech reply in response to the voice communication of the user,

an assistance scheduling server, comprising;

a processor of the assistance scheduling server,

a database of the assistance scheduling server, to store;

an initiation condition associated with an assistance action of an assistance application of the voice controlled assistance service;

one or more voice inputs required as voice inputs for execution of the assistance action, the one or more voice inputs conforming to the voice command protocol of the assistance application;

a memory of the assistance scheduling server, comprising computer-executable instructions that when executed on the processor cause the processor to;

determine occurrence of the initiation condition associated with the assistance action of the assistance application of the voice controlled assistance service;

at least one of (i) transmit an invocation command of the assistance application and the one or more voice inputs required for execution of the assistance action and (ii) transmit an action instruction for the assistance device to initiate the assistance action through the invocation command of the assistance application and the one or more voice inputs required for execution of the assistance action.

7. The system of claim 5, further comprising:

the memory of the assistance device further comprising computer-executable instructions that when executed on the processor cause the processor to;

recognize a custom command word set by the user associated with the assistance application of the voice controlled assistance service and further associated with the assistance action of the assistance application, the assistance action conforming to the voice command protocol of the assistance application;

initiate at least one of a verbal confirmation of the user prior to execution of the assistance application and an input request for one or more additional verbal inputs conforming the voice command protocol before execution of the assistance application.

8. A method comprising:

generating with a speaker a verbal status query to a user at least one of at a predetermined time, at a random time, upon a sensing event in which a presence of the user is sensed, and upon a threshold inactivity of a device of the user;

initiating a countdown timer set to a time length;

determining occurrence of an event of concern that is at least one of;

(i) a negative response event that is a negative response from the user to the verbal status query, the negative response received by a microphone, and (ii) a non-response event that is a failure to receive by the microphone a recognizable response from the user before expiration of the countdown timer;

referencing a safety profile specifying a condition under which one or more events of concern define an adverse event requiring execution of an emergency action;

determining occurrence of the adverse event based on the condition of the safety profile and occurrence of the event of concern;

determining the emergency action requires a notification to at least one of a device of a caregiver of the user and a device of an emergency service;

generating an alert message for at least one of the emergency service and the device of the caregiver of the user, the alert message comprising a log comprising at least one of the adverse event and the one or more events of concern;

repeating the verbal status query to the user and increasing a volume of the verbal status query from the speaker following determination of the non-response event of the user;

upon occurrence of the event of concern, transmitting a status query to the device of the user;

generating a verbal warning to the user with the speaker that at least one of;

(i) the user has failed to provide the recognizable response, (ii) the caregiver has been alerted, and (iii) the emergency service has been alerted;

upon determination of at least one of the event of concern and the adverse event, automatically re-routing a destination of a set of voice communications of the user from a first voice controlled assistance service that provides a general assistance to a second voice controlled assistance service having automated emergency assistance service;

re-routing a voice command of the user from a speech recognition engine providing a general speech recognition to an emergency speech recognition engine specialized in an emergency speech; and

increasing a recognition sensitivity to an emergency word of the user.

9. The method of claim 8, further comprising:

generating the verbal status query to the user with the speaker upon detection of at least one of a sound spike and a vibration spike.

10. The method of claim 8, further comprising:

decreasing the time length to which the countdown timer is set based on a previous occurrence of one or more events of concerns within a predetermined timeframe;

adjusting the condition under which one or more events of concern define the adverse event requiring execution of an emergency action based on the previous occurrence of one or more events of concern within the predetermined timeframe; and

adjusting a length of time of the countdown timer based on the previous occurrence of one or more events of concern within the predetermined timeframe.

11. The method of claim 8, further comprising:

upon occurrence of the adverse event, establishing a first audio feed between an assistance device having the speaker and the microphone and at least one of;

(i) the emergency service and (ii) the device of the caregiver;

upon occurrence of the adverse event, establishing a video feed between the assistance device further having a video camera and at least one of;

(i) the emergency service and (ii) the device of the caregiver; and

upon occurrence of the adverse event, establishing a second audio feed between a sensing pod communicatively coupled to the assistance device and the device of the caregiver, the sensing pod having a second speaker and a second microphone.

12. The method of claim 8, further comprising:

determining an incorrect answer by the user to at least one of a cognitive puzzle, a cognitive game, and a trivia,

wherein the verbal status query is at least one of the cognitive puzzle, the cognitive game, and the trivia, and

wherein the negative response event is the incorrect answer by the user to at least one of the cognitive puzzle, the cognitive game, and the trivia.

13. The method of claim 8, further comprising:

storing an away message of the user, wherein the away message is an audio recording of the user;

associating an expected inactivity time with the away message of the user;

adjusting the condition under which one or more events of concern define the adverse event requiring execution of the emergency action based the expected inactivity time;

suspending the verbal status query until expiration of the expected inactivity time;

communicating the audio recording of the user to the device of the caregiver at least one of;

(i) at recordation of the audio recording of the user, (ii) upon determination of the non-response event, and (iii) upon expiration of the expected inactivity time;

detecting at least one of a particulate at a particulate sensor and a chemical at a chemical sensor;

determining an adverse event requiring execution of the emergency action defined by the detection of at least one of the particulate at the particulate sensor and the chemical at the chemical sensor;

converting a speech response of an assistance service to a textual message with a speech-text converter for display on a graphical user interface; and

populate the graphical user interface with the textual message converted with the speech-text converter for presentation to the user,

wherein the sensing event utilizes a presence sensor to determine a presence of the user within an immediate environment of the presence sensor;

wherein the particulate sensor is a smoke detector and wherein the chemical sensor is at least one of a carbon dioxide sensor and a carbon monoxide sensor,

wherein the recognizable response failing to occur is determined by at least one of detecting a sound above an environment threshold, failure of a speech recognition engine to recognize one or more words spoken by the user, and failure of a voice assistance service to generate a response,

wherein the device of the caregiver is at least one of a mobile device of the caregiver and a wearable device of the caregiver,

wherein the presence sensor is at least one of a motion sensor, an infrared sensor, a proximity sensor, an RFID sensor, a noise sensor, a Bluetooth sensor, and an ultrasonic frequency sound sensor,

wherein the device of the user is at least one of a mobile device of the user, a wearable device of the user, and a connected household device of the user, and

wherein the threshold inactivity is determined based on at least one of a GPS coordinate of the device, a network traffic of the device of the user, an accelerometer data of the device of the user, a microphone of the device of the user, and a video feed of the device of the user.