AUGMENTED REALITY SYSTEM FOR DISPLAYING PATIENT DATA

• US 20180300919A1
• Filed: 02/22/2018
• Published: 10/18/2018
• Est. Priority Date: 02/24/2017
• Status: Active Grant

• Current Assignee: Masimo Corporation
• Original Assignee: Masimo Corporation
• Inventors: Muhsin, Bilal, Ahmed, Omar, Kiani, Massi Joe E.
• Granted Patent: US 11,024,064 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 5/0002 Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network A61B5/07 takes precedence; transmission systems for measured values G08C; transmission H04B; transmission of digital information H04L; wireless communication networks H04W
• CPC Class Codes: A61B 5/743 Displaying an image simultaneously with additional graphical information, e.g. symbols, charts, function plots
• CPC Class Codes: A61B 5/7435 Displaying user selection data, e.g. icons in a graphical user interface
• CPC Class Codes: A61B 5/744 Displaying an avatar, e.g. an animated cartoon character
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: G16H 50/50 for simulation or modelling of medical disorders
• CPC Class Codes: H04L 63/083 using passwords cryptographic mechanisms or cryptographic arrangements for entity authentication using a predetermined code H04L9/3226

Abstract

System and methods are provided for augmented reality displays for medical and physiological monitoring. Augmented reality user interfaces are virtually pinned to a physical device, a location, or to a patient. An augmented reality position determination process determines the presentation of user interfaces relative to reference positions and reference objects. Detection of gestures causes the augmented reality users interfaces to be updated, such as pinning a user interface to a device, location, or patient. Looking away from an augmented reality user interface causes the user interface to minimize or disappear in an augmented reality display. An augmented reality gesture detection process determines gestures based on captured image data and computer vision techniques performed on the image data.

Claims

1. A method for presenting augmented reality data from a medical monitoring device, the method comprising:

under control of a hardware processor,

receiving physiological monitoring data comprising physiological parameter values associated with a patient from a monitoring hub;

accessing user interface configuration data;

generating, from the physiological monitoring data, a plurality of augmented reality objects according to the user interface configuration data;

receiving user interaction data from a user input device of an augmented reality device, wherein the user interaction data comprises an indication of an interaction to virtually pin the plurality of augmented reality objects to a reference object;

determining a reference position based at least in part the user interaction data; and

causing presentation of the plurality of augmented reality objects in an augmented reality display, wherein the plurality of augmented reality objects are presented relative to the reference position.

2. The method of claim 1, further comprising:

identifying, from the user interaction data, the reference object;

determining the reference position for the reference object; and

calculating a positional offset from the reference position, wherein the plurality of augmented reality objects are presented relative to the reference position according to the positional offset.

3. The method of claim 1, further comprising:

determining, from the user interface configuration data, whether to present a direct overlay;

identifying, from the user interface configuration data, the reference object;

calculating the reference position for the reference object, wherein an overlay object of the plurality of augmented reality objects is presented at the reference position.

4. The method of claim 1, wherein the reference object is the patient.

5. The method of claim 1, further comprising:

detecting a gesture from the user interaction data; and

generating the user interface configuration data based at least in part on the gesture.

6. The method of claim 5, wherein the user input device comprises a camera.

7. The method of claim 6, wherein the user input data comprises image data, and wherein detecting the gesture from the user input data further comprises:

determining color histogram data from the image data;

locating a search window in the image data according to the color histogram data; and

identifying a plurality of positions of the search window in the image data.

8. The method of claim 1, wherein the plurality of augmented reality objects are presented in a first arrangement, the method further comprising:

receiving second user interaction data from the user input device, the second user interaction data indicating a second arrangement of the plurality of augmented reality objects;

generating second user interface configuration data from the second user interaction data; and

causing presentation, in the augmented reality display, of the plurality of augmented reality objects in the second arrangement according to the second user interface configuration data.

9. The method of claim 1, further comprising:

receiving hub user interaction data from the monitoring hub;

determining a hub user interface configuration from the hub user interaction data, wherein the user interface configuration data is indicative of the hub user interface configuration; and

causing presentation of a user interface on a display of the monitoring hub according to the hub user interface configuration.

10. The method of claim 9, wherein hub user interface configuration comprises a plurality of user interface elements, wherein each element of the plurality of user interface elements comprises a physiological parameter value, and wherein each element of the plurality of user interface elements corresponds to an object from the plurality of augmented reality objects.

11. The method of claim 9, wherein the hub user interaction data is indicative of at least one addition, removal, or rearrangement of a user interface element.

12. An augmented reality device configured to present physiological data, the system comprising:

a memory device configured to store instructions; and

a hardware processor configured to execute the instructions to;

receive physiological monitoring data comprising physiological parameter values associated with a patient from a patient monitor;

access user interface configuration data;

generate, from the physiological monitoring data, a plurality of augmented reality objects according to the user interface configuration data;

determine that a clinician wearing the augmented reality device is looking toward the patient monitor, and

in response to said determination, cause presentation of the plurality of augmented reality objects in an augmented reality display in a vicinity of the patient monitor.

13. The augmented reality device of claim 12, wherein the hardware processor is further configured to:

receive user interaction data from a user input device of an augmented reality device; and

generate the user interface configuration data from the user interaction data.

14. The augmented reality device of claim 12, wherein the hardware processor is further configured to:

receive user interaction data from a user input device of an augmented reality device;

identify, from the user interaction data, a reference object;

determine a reference position for the reference object; and

calculate a positional offset from the reference position, wherein the plurality of augmented reality objects are presented relative to the reference position according to the positional offset.

15. The augmented reality device of claim 12, wherein the hardware processor is further configured to:

determine, from the user interface configuration data, whether to present a direct overlay;

identify, from the user interface configuration data, a reference object; and

calculating a reference position for the reference object, wherein an overlay object of the plurality of augmented reality objects is presented at the reference position.

16. The augmented reality device of claim 12, wherein the hardware processor is further configured to:

receive user input data from a user input device;

detect a gesture from the user input data; and

generate the user interface configuration data based at least in part on the gesture.

17. The augmented reality device of claim 16, wherein the user input device comprises a camera.

18. The augmented reality device of claim 16, wherein the user input data comprises image data, and wherein detecting the gesture from the user input data further comprises:

determining color histogram data from the image data;

locating a search window in the image data according to the color histogram data; and

identifying a plurality of positions of the search window in the image data.

19. The augmented reality device of claim 12, wherein the hardware processor is further configured to:

receive monitor user interaction data from the patient monitor;

determine a monitor user interface configuration from the monitor user interaction data, wherein the user interface configuration data is indicative of the monitor user interface configuration; and

cause presentation of a user interface on a display of the patient monitor according to the monitor user interface configuration.

20. The augmented reality device of claim 12, wherein at least some of the plurality of augmented reality objects correspond to the monitor user interface configuration.

21. A system for presenting augmented reality data from a medical monitoring device, the system comprising:

an augmented reality device comprising a hardware processor, memory, a display, and a wireless device configured to communicate with a medical monitoring device in communication with a patient, the hardware processor of the augmented reality device configured to;

identify a user interaction with the augmented reality device by a clinician wearing the augmented reality device, the user interaction comprising a gesture;

determine that the user interaction is an instruction to virtually pin a user interface of the medical monitoring device to a virtual location separate from the medical monitoring device;

wirelessly obtain patient data depicted in the user interface from the medical monitoring device; and

output for presentation to the clinician, in the augmented reality device, a display of the user interface pinned to the virtual location.

22. The system of claim 21, wherein the instruction is to virtually pin the user interface outside of and next to a hospital room of the patient so that the clinician can subsequently view the user interface while walking past the hospital room and without entering the hospital room.

23. The system of claim 21, wherein the display of the user interface pinned to the virtual location is viewable only when the display'"'"'s field of view includes the virtual location, and wherein the virtual location does not obscure the patient.

24. The system of claim 21, wherein the augmented reality device is further operable to detect that the clinician has moved his or her head away from looking at the medical monitoring device and to reduce a size or content of the user interface at the virtual location.

25. The system of claim 24, wherein the augmented reality device further comprises a camera or a movement sensor configured to detect that the clinician has moved his or her head away from looking at the medical monitoring device.

26. The system of claim 21, wherein the user interface is configurable via a device different from the augmented reality device.