Healthcare systems and methods using voice inputs

• US 10,347,255 B1
• Filed: 09/14/2018
• Issued: 07/09/2019
• Est. Priority Date: 03/08/2018
• Status: Active Grant

• Current Assignee: CES Advancements, LLC
• Original Assignee: CES Advancements, LLC
• Inventors: Paul, Martin G., Subramanian, PV
• Primary Examiner(s): Vo, Huyen X
• Application Number:
  US16/132,134
• Time in Patent Office: 298 Days
• Field of Search: 704 1- 10, 704230-257, 704270-275
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G10L 15/07 to the speaker
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 40/20 for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
• CPC Class Codes: G16H 40/63 for local operation

Abstract

A voice-enabled digital communications assistant powered in part using tailored machine learning models and other algorithms is used to engage with and control one or more healthcare devices or instruments such that a user is able to control the devices or instruments using natural language, conversational-like, voice commands. A command processor processes the audible instructions, while a context-aware processor monitors the present states and conditions of all devices and instruments, as well as the environment, for situational awareness purposes, including situations where executing commands may be incompatible with or conflict with the present states or conditions of devices and instruments as well as their expected future states. In addition to speech responses by the digital assistant, a separate notification engine provides audible or visual feedback to the user.

Claims

1. A voice-enabled digital communications assistant system comprising:

at least one healthcare device or instrument each characterized by a present state or condition and each adapted to performing a plurality of functions or actions in at least an operating room;

at least one controller associated with each of the at least one device or instrument for causing the at least one device or instrument to perform one or more of the plurality of functions or actions in its present state or condition;

at least one audio input device for receiving real-time human-generated speech in at least the operating room and for outputting a signal representing the speech;

at least one knowledge base containing a list of mappings or decision trees relating at least the present state or condition, a plurality of future states or conditions, and the plurality of functions and actions, and further containing a list of capabilities of each of the at least one device or instrument;

a processor adapted to executing a digital media-containing software, the software being adapted to

using natural language processing or other machine learning algorithms to extract semantic meaning from the outputted signal from the audio input device,

identifying which one of the at least one device or instrument the speech is directed to based on the present state or condition, the capabilities, and the extracted meaning,

identifying from the at least one knowledge base one or more of the plurality of future states or conditions of the identified or instrument,

selecting the present or one of the plurality of future states or conditions to place the identified device or instrument in, and

generating a control signal to send to the identified device or instrument or to its at least one controller;

wherein the control signal includes information useful for causing the identified device or instrument, systems, or subsystems to perform at least one of the plurality of functions or take at least one of the plurality of actions.

2. The system according to claim 1, wherein the control signal is sent to a notification device when the at least one knowledge base contains no mapping or decision tree relating the extracted meaning from the speech to the plurality of functions or actions, the present states or conditions, the plurality of future states or conditions, the capabilities, or a combination thereof.

3. The system according to claim 1, wherein at least one of the plurality of functions is a surgical function adapted to being performed in at least the operating room.

4. The system according to claim 1, wherein the at least one audio input device includes a far-field microphone.

5. The system according to claim 1, wherein the processor and software are part of a remote cloud server and receive the outputted audio signal from the at least one audio input device via an API.

6. A voice-enabled digital communications assistant system comprising:

at least one healthcare device or instrument characterized by a present state or condition and adapted to performing one or more of a plurality of operating room functions or actions in at least an operating room;

at least one audio input device within at least the operating room for receiving and processing real-time human-generated speech from inside or outside the operating room and for outputting a speech signal in response to the speech;

a context engine for receiving the speech signal, an information about the present state or condition, and an information from one or more knowledge bases, wherein the information from the one or more knowledge bases includes a plurality of future states or conditions of the at least one device or instrument, its capabilities, and a work flow of a healthcare procedure to be performed in at least the operating room, and wherein the context engine is adapted to,

extracting semantic meaning from the speech signal;

identifying which one of the at least one device or instrument the speech is directed to based on at least the information about the present state or condition, the capabilities, the work flow, and the extracted meaning;

identifying one or more of the plurality of future states or conditions of the identified device or instrument,

selecting the present or one of the identified future states or conditions to place the identified device or instrument in, and

generating a context signal, based on the selection and the list of capabilities; and

a command processor for receiving an output from the context signal and outputting a command signal,

wherein the command signal includes information useful for causing the identified device or instrument to remain in the present state or condition or to perform at least one of the plurality of functions or actions.

7. The system according to claim 6, wherein the speech comprises a verbal command directed to causing a change in the present state or condition.

8. The system according to claim 7, wherein the command signal comprises a notification signal when the verbal command cannot be achieved by any of the future states or conditions, or when the verbal command exceeds the capabilities.

9. A voice-enabled healthcare method comprising:

providing one or more healthcare instruments or devices each characterized by one or more present states, conditions, and capabilities, and a plurality of future states and conditions, and wherein each is adapted to performing one or more of a plurality of functions or actions;

monitoring and storing an information about the present state or condition of each of the one or more instruments or devices;

receiving in an input device a real-time human-generated speech;

using natural language processing or other machine learning algorithms, extract semantic meaning from the speech;

identifying which of the one or more devices or instruments the speech is directed to based on their respective present states or conditions, their respective capabilities, and the extracted meaning;

identifying one or more of the plurality of future states or conditions of the identified one or more devices or instruments;

selecting the present or one of the identified one or more plurality of future states or conditions to place the identified ones of the devices or instruments in; and

generating a control signal, based on the selection, to send to the identified ones of the devices or instruments for performing the at least one of the plurality of functions or actions.

10. The method according to claim 9, wherein the steps of extracting semantic meaning from the speech and generating a control signal comprise sending information to a server using an API.

11. The method according to claim 9, further comprising updating the stored present state or condition information with the future state or condition information when the at least one of the plurality of functions or actions is performed.

12. The method according to claim 9, wherein the speech is generated by a plurality of users, and wherein extracting meaning from the speech comprises differentiating the speech between the users using speech recognition patterns or a location beacon attached to each of the plurality of users.

13. The method according to claim 9, wherein the speech comprises one or more verbal commands directed to causing a change in the present state or condition.

14. The method according to claim 9, wherein performing the at least one of the plurality of functions or actions comprises performing the function or action in a hospital operating room.