Audio, visual and device data capturing system with real-time speech recognition command and control system

• US 8,502,876 B2
• Filed: 09/12/2006
• Issued: 08/06/2013
• Est. Priority Date: 09/12/2006
• Status: Active Grant

• Current Assignee: Storz Endoskop Produktions GmbH (Karl Storz Se & Co. Kg)
• Original Assignee: Storz Endoskop Produktions GmbH (Karl Storz Se & Co. Kg)
• Inventors: Wang, Gang, Contolini, Matteo, Zheng, Chengyi, Stiller, Heinz-Werner
• Primary Examiner(s): AGGARWAL, YOGESH K
• Application Number:
  US11/519,315
• Publication Number:
  US 20080062280A1
• Time in Patent Office: 2,520 Days
• Field of Search: 348/207.1, 348/222.1, 348373-375, 704/725
• US Class Current: 348/222.1
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing

Abstract

An audio, visual and device data capturing system including an audio recorder for recording audio data, at least one visual recorder for recording visual data, at least one device data recorder for receiving device data from at least one device in communication with the system, a speech recognition module for interpreting the audio data, a transcript module for generating transcript data from the interpreted audio data, a data capturing module for generating a data record including at least a portion of each of the audio data, the transcript data, the visual data and the device data, and at least one storage device for storing the data record.

Claims

1. An audio and visual data capturing system, comprising:

an audio recorder for recording audio data including a plurality of user utterances;

at least one visual recorder for recording visual data;

a speech recognition module for interpreting the audio data;

one or more medical devices in communication with the system;

a discriminator for distinguishing, based on a recognition grammar, between interpreted audio data for transcription and interpreted audio data for controlling the one or more medical devices in communication with the system;

a transcript module for generating transcript data from the interpreted audio data identified by the discriminator for transcription;

at least one device data recorder for receiving device data, including at least one of a device status and a device setting, from at least one of the medical devices in communication with the system;

a data capturing module for generating a data record including at least a portion of each of the audio data, the transcript data, the visual data, and the device data; and

at least one storage device for storing the data record.

2. The system according to claim 1, wherein said data capturing module synchronizes the audio data, the transcript data, the visual data and the device data in the data record.

3. The system according to claim 1, wherein the audio data includes one or more voice annotations corresponding to one or more portions of the visual data.

4. The system according to claim 1, wherein said at least one storage device includes a plurality of data records, wherein the system comprises software for querying the plurality of data records based on the transcript data.

5. The system according to claim 1, wherein said transcript module generates the transcript data continuously while said audio recorder records the audio data.

6. The system according to claim 1, wherein said at least one storage device includes a database, wherein the data record is stored in a location of the database based on at least a portion of the transcript data.

7. The system according to claim 1, wherein said at least one storage device includes a medical records database.

8. The system according to claim 1, wherein said at least one storage device includes a portable media drive.

9. The system according to claim 1, wherein said at least one visual recorder includes a video image recorder and a still image recorder, wherein the visual data includes both still image data and video image data.

10. The system according to claim 1, wherein the one or more medical devices in communication with the system further include an endoscope, and wherein the endoscope comprises said at least one visual recorder.

11. The system according to claim 1, wherein the recognition grammar is a dynamic recognition grammar updated upon at least one of a device status change and a command menu change.

12. The system according to claim 1, wherein the data record is searchable.

13. The system according to claim 1, wherein the data record, including at least one of the transcript data and the device data in the data record, is searchable using a search parameter.

14. The system according to claim 1, wherein the one or more medical devices in communication with the system include at least one of an insufflator, a light source, a vacuum pump, a video display, and a surgical table.

15. An audio and visual data capturing system with speech recognition and control, comprising:

an audio recorder for recording audio data including a plurality of user utterances;

at least one visual recorder for recording visual data;

a speech recognition module for interpreting the audio data;

one or more medical devices in communication with the system;

a discriminator for distinguishing, based on a recognition grammar, between interpreted audio data for transcription and interpreted audio data for controlling the one or more medical devices in communication with the system;

a controller for generating a control command based on at least a portion of the interpreted audio data identified by the discriminator for controlling the one or more devices in communication with the system;

a transcript module for generating transcript data from the interpreted audio data identified by the discriminator for transcription;

at least one device data recorder for receiving device data, including at least one of a device status and a device setting, from at least one of the medical devices in communication with the system;

a data capturing module for generating a data record including at least a portion of each of the audio data, the visual data, the transcript data, and the device data; and

at least one storage device for storing the data record.

16. The system according to claim 15, wherein the recognition grammar is updated upon at least one of a device status change and a command menu change.

17. The system according to claim 15, wherein the speech recognition module includes a transcript module for generating the transcript data.

18. The system according to claim 15, wherein said data capturing module synchronizes the audio data, the visual data and the transcript data.

19. The system according to claim 15, wherein said controller further sends the control command to at least one of the medical devices operable by the system.

20. The system according to claim 15, wherein the one or more medical devices in communication with the system include at least one of an insufflator, a light source, a vacuum pump, a video display, and a surgical table.

21. A method of processing audio, visual and device data, comprising the steps of:

recording audio data, the audio data including one or more user utterances;

receiving device data from at least one medical device, including at least one of an insufflator, a light source, a vacuum pump, a video display, and a surgical table, wherein the device data including at least one of a device status and a device setting;

interpreting the one or more user utterances and discriminating, based on a recognition grammar, between user utterances for transcription and user utterances for controlling the at least one medical device;

generating transcript data from the user utterances identified for transcription;

generating a data record including at least a portion of each of the audio data, the transcript data, and the device data; and

storing the data record.

22. The method according to claim 21, further comprising the step of:

synchronizing the audio data and the transcript data in the data record.

23. The method according to claim 21, wherein the data record further includes the visual data, wherein the method further includes the step of synchronizing the audio data, the transcript data, and the visual data in the data record.

24. The method according to claim 23, further comprising the steps of:

receiving a search request including at least one search parameter;

querying the data record, including at least one of the transcript data and the device data in the data record, for the at least one search parameter; and

providing at least a portion of the data record corresponding to the at least one search parameter.

25. The method according to claim 21, wherein the method further includes the step of synchronizing the audio data, the transcript data, and the device data in the data record.

26. The method according to claim 21, wherein said step of discriminating including querying the recognition grammar, wherein the method further comprises a step of updating the recognition grammar upon one of a device status change and a command menu change.

27. The method according to claim 21, further comprising the step of:

sending the user utterances intended as control commands to a command controller for execution.

28. The method according to claim 27, further comprising the step of:

sending a control command to at least one of the medical devices operable by the control command.

29. The method according to claim 21, wherein at least one of the user utterances corresponds to a portion of the visual data.

30. The method according to claim 21, wherein the data record is stored in a database location based on the transcript data.

31. The method according to claim 21, further comprising the step of:

generating a control command based on the interpreted audio data and sending the control command to at least one of the medical devices operable by the control command.

32. The method according to claim 21, further comprising the steps of:

receiving a search request including at least one search parameter;

querying the data record for the at least one search parameter; and

providing at least a portion of the data record corresponding to the at least one search parameter.