System and method for accessing and annotating electronic medical records using a multi-modal interface

• US 7,499,862 B1
• Filed: 04/23/2007
• Issued: 03/03/2009
• Est. Priority Date: 06/14/2002
• Status: Expired due to Fees

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Bangalore, Srinivas, Johnston, Michael, Blewett, Charles Douglas
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US11/788,890
• Time in Patent Office: 680 Days
• Field of Search: 704270-275, 600/300, 600/484, 600/523, 600/301, 707/104.1, 709/206
• US Class Current: 704/275
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 15/00 ICT specially adapted for medical reports, e.g. generation or transmission thereof
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A system and method of exchanging medical information between a user and a computer device is disclosed. The computer device can receive user input in one of a plurality of types of user input comprising speech, pen, gesture and a combination of speech, pen and gesture. The method comprises receiving information from the user associated with a medical condition and a bodily location of the medical condition on a patient in one of a plurality of types of user input, presenting in one of a plurality of types of system output an indication of the received medical condition and the bodily location of the medical condition, and presenting to the user an indication that the computer device is ready to receive further information. The invention enables a more flexible multi-modal interactive environment for entering medical information into a computer device. The medical device also generates multi-modal output for presenting a patient'"'"'s medical condition in an efficient manner.

Claims

1. A method of exchanging medical information between a user and a computer device arranged to receive input from the user in a plurality of modes of user input, the method comprising:

receiving from the user a synchronized combination of a plurality of user inputs associated with a medical condition and a bodily location of the medical condition on a patient, the plurality of user inputs including user inputs in at least two user input modes;

using a multi-modal finite state transducer to combine the plurality of user inputs and to determine an indication of the received medical condition and the bodily location of the medical condition;

presenting to the user an output of the indication; and

presenting to the user an indication that the computer device is ready to receive further information.

2. The method of claim 1, wherein the plurality of user inputs includes speech recognition.

3. The method of claim 1, wherein the plurality of user inputs includes gesture.

4. The method of claim 1, wherein the plurality of user inputs includes a pen and tablet.

5. The method of claim 1, wherein the computer device comprises an electronic medical chart.

6. The method of claim 1, wherein the step of presenting to the user an output the indication includes presenting through graphics.

7. The method of claim 1, wherein the step of presenting to the user an output of the indication includes presenting through synthesized speech.

8. The method of claim 1, wherein the step of presenting to the user an output of the indication includes presenting through text.

9. The method of claim 1, wherein the step of presenting to the user an output of the indication further comprises presenting a graphical indication on a patient diagram of the location of the bodily condition.

10. The method of claim 1, wherein the step of presenting to the user an output of the indication further comprises presenting a graphical indication in a tree browser of annotations.

11. The method of claim 1, wherein the medical condition relates to at least one of a wound and a bone fracture.

12. The method of claim 1, wherein the medical condition relates to medicine administered to the patient.

13. The method of claim 1, the step of presenting to the user an output of the indication further comprises highlighting a patient image on a display screen indicating the location of the bodily condition and indicating the bodily condition.

14. The method of claim 1, further comprising the step of:

presenting the user with an option to request a summary of received user information.

15. The method of claim 14, wherein if the user requests a summary of received user information, the method further comprises:

presenting a summary of a patient condition according to the received user information.

16. The method of claim 15, wherein presenting the summary of the patient condition further comprises highlighting on a patient graphic the bodily location and providing synthetic speech to describe the bodily condition associated with the highlighted bodily location.

17. An apparatus for exchanging medical information between the apparatus and a user, the apparatus the apparatus comprising:

(1) a plurality of user interfaces for receiving a synchronized combination of inputs from the user associated with a medical condition and a bodily location of the medical condition on a patient, the interfaces including interfaces capable of receiving user inputs in at least two user input modes;

(2) a multi-modal recognition module using a finite-state machine to build a single meaning representation from user inputs received in the at least two user input modes; and

(3) at least one user interface for presenting an indication of the received medical condition and the bodily location of the medical condition.

18. The apparatus of claim 17, further comprising:

(4) at least one user interface for presenting to the user an indication that the computer device is ready to receive further information.

19. The apparatus of claim 17, wherein the user interface for presenting an indication of the received medical condition and the bodily location of the medical condition is further for presenting a hierarchical summary of the received information associated with the medical condition and the bodily location of the medical condition.

20. The apparatus of claim 17, wherein the plurality of user interfaces for receiving inputs comprises at least one interface selected from the group consisting of voice recognition, pen and tablet and gesture.