Semiautomated relay method and apparatus

• US 10,389,876 B2
• Filed: 02/26/2015
• Issued: 08/20/2019
• Est. Priority Date: 02/28/2014
• Status: Active Grant

• Current Assignee: Ultratech Incorporated (Veeco Instruments Incorporated)
• Original Assignee: Ultratech Incorporated (Veeco Instruments Incorporated)
• Inventors: Engelke, Robert M., Colwell, Kevin R., Engelke, Christopher R.
• Primary Examiner(s): Phan, Joseph T
• Application Number:
  US14/632,257
• Publication Number:
  US 20170201613A1
• Time in Patent Office: 1,636 Days
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 21/10 Transforming into visible information
• CPC Class Codes: H04M 1/2475 for a hearing impaired user
• CPC Class Codes: H04M 1/72478 for hearing-impaired users
• CPC Class Codes: H04M 2201/18 Comparators
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 2201/405 involving speaker-dependent recognition adaptation to speaker for speech recognition G10L15/07
• CPC Class Codes: H04M 3/42391 where the subscribers are hearing-impaired persons, e.g. telephone devices for the deaf
• CPC Class Codes: H04M 3/53366 Message disposing or creating aspects
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements
• CPC Class Codes: H04W 4/16 Communication-related supplementary services, e.g. call-transfer or call-hold

Abstract

A system for providing voice-to-text captioning service comprising a relay processor that receives voice messages generated by a hearing user during a call, the processor programmed to present the voice messages to a first call assistant, a call assistant device used by the first call assistant to generate call assistant generated text corresponding to the hearing user'"'"'s voice messages, the processor further programmed to run automated voice-to-text transcription software to generate automated text corresponding to the hearing user'"'"'s voice messages, use the call assistant generated text, the automated text and the hearing user'"'"'s voice messages to train the voice-to-text transcription software to more accurately transcribe the hearing user'"'"'s voice messages to text, determine when the accuracy exceeds an accuracy requirement threshold, during the call and prior to the automated text exceeding the accuracy requirement threshold, transmitting the call assistant generated text to the assisted user'"'"'s device for display to the assisted user and subsequent to the automated text exceeding the accuracy requirement threshold, transmitting the automated text to the assisted user'"'"'s device for display to the assisted user.

Claims

1. A system for providing voice-to-text captioning service for a hearing impaired assisted user using an assisted user'"'"'s device to communicate with a hearing user that uses a hearing user'"'"'s device, the system comprising:

a relay processor that receives voice messages generated by the hearing user during a call between the assisted user and the hearing user, the relay processor programmed to present the voice messages to a first call assistant;

a call assistant device used by the first call assistant to generate call assistant generated text corresponding to the hearing user'"'"'s voice messages;

the relay processor further programmed to;

run automated voice-to-text transcription software to generate automated text corresponding to the hearing user'"'"'s voice messages;

use the call assistant generated text, the automated text and the hearing user'"'"'s voice messages to train the voice-to-text transcription software to more accurately transcribe the hearing user'"'"'s voice messages to text;

determine when the accuracy of the automated text exceeds an accuracy requirement threshold;

during the call and prior to the automated text exceeding the accuracy requirement threshold, transmitting the call assistant generated text to the assisted user'"'"'s device for display to the assisted user; and

subsequent to the automated text exceeding the accuracy requirement threshold, transmitting the automated text to the assisted user'"'"'s device for display to the assisted user.

2. The system of claim 1 wherein, subsequent to the automated text exceeding the accuracy requirement threshold, the relay processor stops presenting the hearing user'"'"'s voice messages to the first call assistant so that the first call assistant can transcribe a different hearing user'"'"'s voice messages.

3. The system of claim 2 further including presenting the text received at the assisted user'"'"'s device to the assisted user via a device display screen, when automated text from the relay is presented via the display screen, monitoring for an assistance request via the assisted users device and, when an assistance request is received, presenting the hearing user'"'"'s voice messages to another call assistant to again generate call assistant generated text, transmitting the call assistant generated text to the assisted user'"'"'s device for display and halting transmission of the automated text to the assisted user'"'"'s device.

4. The system of claim 3 wherein the step of presenting the hearing user'"'"'s voice messages to a call assistant after the assistance request has been received includes, determining if the first call assistant is available to handle the hearing user'"'"'s voice messages and, if the first call assistant is available, presenting the hearing user'"'"'s voice messages to the first call assistant to be transcribed.

5. The system of claim 3 wherein an assistance request is initiated by selecting a help button of the assisted user'"'"'s device.

6. The system of claim 2 further including presenting the text received at the assisted user'"'"'s device to the assisted user via a device display screen, when automated text from the relay is presented via the display screen, also presenting a virtual help button via the display that is selectable by a system user to request captioning from a call assistant.

7. The system of claim 2 further including presenting the call assistant generated text received at the assisted user'"'"'s device to the assisted user via a device display screen and presenting an indication of a delay in text transcription on the device display screen.

8. The system of claim 7 wherein the indication of a delay includes a number of seconds indicator indicating the duration of an instantaneous delay.

9. The system of claim 7 wherein the indication of a delay includes a representation of words uttered by a hearing user that has yet to be transcribed to text at the end of a current string of displayed text wherein a length dimension of the representation of words uttered is associated with the duration of a current delay in transcription to text.

10. The system of claim 7 further including the step of presenting a re-sync option via the assisted user'"'"'s device display, detecting selection of the re-sync option and, in response to selection of the re-sync option, skipping transcription of at least a portion of the hearing user'"'"'s voice message so that the transcription process continues on a more current segment of the hearing user'"'"'s voice message.

11. The system of claim 10 wherein, in response to selection of the re-sync option, the system skips to a pre-defined duration period that precedes the time when the re-sync selection was made so that automatically generated text is presented as fill in text for the hearing user'"'"'s voice message that corresponds to the pre-defined duration period up to a current time.

12. The system of claim 11 wherein the fill in text is visually distinguished from other text presented on the assisted user'"'"'s device display.

13. The system of claim 1 further including the steps of, subsequent to the automated text exceeding the accuracy requirement threshold, presenting the automated text and the hearing user'"'"'s voice messages to a second call assistant for correction and, after the second call assistant corrects errors in the automated text, transmitting the corrections to the assisted user'"'"'s device.

14. The system of claim 13 wherein the assisted user'"'"'s device presents the corrections on the assisted user'"'"'s device display screen.

15. The system of claim 14 wherein the assisted user'"'"'s device presents the corrections by replacing errors in the text on the assisted user'"'"'s device display screen with corrected blocks of text.

16. The system of claim 1 further including the steps of, after at least some training of the voice-to-text software, storing a voice model for the hearing user in a database to be used during subsequent communications between the hearing user and the assisted user.

17. The system of claim 16 wherein each stored voice model is associated with a hearing user'"'"'s device identifier in the database so that the next time the hearing user'"'"'s device is linked to the assisted user'"'"'s device to facilitate a communication, the device identifier can be used to identify the voice model associated with the hearing user.

18. The system of claim 17 wherein more than one voice model is associated with at least one of the device identifiers in the database.

19. The system of claim 18 wherein a voice profile for identifying a specific voice is stored in the database for each of the more than one voice models associated with the at least one of the device identifiers.

20. The system of claim 1 wherein the hearing user'"'"'s voice messages received at the relay processor are received from the assisted user'"'"'s device.

21. The system of claim 1 wherein the relay processor trains the voice-to-text transcription software by comparing the call assistant generated text to the automated text to identify errors in the automated text and then adjusts the software to avoid the same error subsequently.

22. The system of claim 1 wherein the relay processor determines when the accuracy of the automated text exceeds an accuracy requirement threshold by comparing the automated text to the call assistant generated text to generate an accuracy value for the automated text and then compares the accuracy value to the accuracy requirement.

23. The system of claim 1 wherein the call assistant generates text by revoicing the hearing user'"'"'s voice messages to voice-to-text transcription software trained to the voice of the call assistant, the call assistant generated text transmitted to the assisted user'"'"'s device and simultaneously presented to the call assistant for manual correction on a display screen, corrections by the call assistant to the call assistant generated text transmitted to the assisted user'"'"'s device for presentation in line in the text presented to the assisted user, the processor using the corrected text to train the software to the voice of the hearing user.

24. The system of claim 1 wherein, while transmitting the automated text to the assisted user'"'"'s device for display, the relay processor monitors transcription accuracy, automatically identifies accuracy degradation and, upon recognizing accuracy degradation of a threshold level, causes a re-link to a call assistant for generation of call assistant generated text to be transmitted to the assisted user'"'"'s device for display to the assisted user.

25. The system of claim 1 wherein, while transmitting the automated text to the assisted user'"'"'s device for display, the relay processor monitors transcription accuracy, automatically identifies accuracy degradation and, upon recognizing accuracy degradation of a threshold level, causes the automated text to be presented to a call assistant for correction.

26. The system of claim 25 wherein the automated text is presented to the call assistant for correction simultaneously with transmission of the automated text to the assisted user'"'"'s device for display, any corrections by the call assistant to the automated text subsequently transmitted to the assisted user'"'"'s device for in line corrections where errors are replaced by corrected text.

27. The system of claim 1 wherein the accuracy requirement threshold is 96% or greater accuracy.

28. The system of claim 27 wherein the accuracy requirement threshold is measured over a preceding transcription period.

29. The system of claim 28 wherein the preceding transcription period is less than one minute.

30. The system of claim 27 wherein the accuracy requirement threshold is measured over a preceding predetermined number of transcribed words.

31. A system for providing voice-to-text captioning service for a hearing impaired assisted user using an assisted user'"'"'s device to communicate with a hearing user that uses a hearing user'"'"'s device, the system comprising:

a relay processor that receives voice messages generated by the hearing user during a call between the assisted user and the hearing user, the relay processor programmed to present the voice messages to a first call assistant;

a call assistant device used by the first call assistant to generate call assistant generated text corresponding to the hearing user'"'"'s voice messages;

the relay processor further programmed to;

run automated voice-to-text transcription software to generate automated text corresponding to the hearing user'"'"'s voice messages;

use the call assistant generated text, the automated text and the hearing user'"'"'s voice messages to train the voice-to-text transcription software to more accurately transcribe the hearing user'"'"'s voice messages to text;

determine when the accuracy of the automated text exceeds an accuracy requirement threshold;

during the call and prior to the automated text exceeding the accuracy requirement threshold, transmitting the call assistant generated text to the assisted user'"'"'s device for display to the assisted user;

subsequent to the automated text exceeding the accuracy requirement threshold, transmitting the automated text to the assisted user'"'"'s device for display to the assisted user; and

after at least some training of the voice-to-text transcription software, storing a voice model for the hearing user in a database to be used during subsequent communications between the hearing user and the assisted user.

32. The system of claim 31 wherein, subsequent to the automated text exceeding the accuracy requirement threshold, the relay processor stops presenting the hearing user'"'"'s voice messages to the first call assistant so that the first call assistant can transcribe a different hearing user'"'"'s voice messages.

33. The system of claim 31 further including presenting the text received at the assisted user'"'"'s device to the assisted user via a device display screen, when automated text from the relay is presented via the display screen, monitoring for an assistance request via the assisted users device and, when an assistance request is received, presenting the hearing user'"'"'s voice messages to another call assistant to again generate call assistant generated text, transmitting the call assistant generated text to the assisted user'"'"'s device for display and halting transmission of the automated text to the assisted user'"'"'s device.

34. The system of claim 33 wherein an assistance request is initiated by selecting a help button of the assisted user'"'"'s device.

35. The system of claim 31 wherein the hearing user'"'"'s voice messages received at the relay processor are received from the assisted user'"'"'s device.

36. The system of claim 31 wherein the relay processor trains the voice-to-text transcription software by comparing the call assistant generated text to the automated text to identify errors in the automated text and then adjusts the software to avoid the same error subsequently.

37. The system of claim 36 wherein the relay processor determines when the accuracy of the automated text exceeds an accuracy requirement threshold by comparing the automated text to the call assistant generated text to generate an accuracy value for the automated text and then compares the accuracy value to the accuracy requirement.

38. The system of claim 37 wherein the call assistant generates text by revoicing the hearing user'"'"'s voice messages to voice-to-text transcription software trained to the voice of the call assistant, the call assistant generated text transmitted to the assisted user'"'"'s device and simultaneously presented to the call assistant for manual correction on a display screen, corrections by the call assistant to the call assistant generated text transmitted to the assisted user'"'"'s device for presentation in line in the text presented to the assisted user, the processor using the corrected text to train the software to the voice of the hearing user.

39. The system of claim 38 wherein, while transmitting the automated text to the assisted user'"'"'s device for display, the relay processor monitors transcription accuracy, automatically identifies accuracy degradation and, upon recognizing accuracy degradation of a threshold level, causes a re-link to a call assistant for generation of call assistant generated text to be transmitted to the assisted user'"'"'s device for display to the assisted user.

40. The system of claim 38 wherein, while transmitting the automated text to the assisted user'"'"'s device for display, the relay processor monitors transcription accuracy, automatically identifies accuracy degradation and, upon recognizing accuracy degradation of a threshold level, causes the automated text to be presented to a call assistant for correction.

41. The system of claim 40 wherein the automated text is presented to the call assistant for correction simultaneously with transmission of the automated text to the assisted user'"'"'s device for display, any corrections by the call assistant to the automated text subsequently transmitted to the assisted user'"'"'s device for in line corrections where errors are replaced by corrected text.

42. The system of claim 38 further including presenting the call assistant generated text received at the assisted user'"'"'s device to the assisted user via a device display screen and presenting an indication of a delay in text transcription on the device display screen.

43. The system of claim 42 wherein the indication of a delay includes a number of seconds indicator indicating the duration of an instantaneous delay.

44. The system of claim 42 wherein the indication of a delay includes a representation of words uttered by a hearing user that has yet to be transcribed to text at the end of a current string of displayed text wherein a length dimension of the representation of words uttered is associated with the duration of a current delay in transcription to text.

45. The system of claim 31 further including the step of presenting a re-sync option via the assisted user'"'"'s device display, detecting selection of the re-sync option and, in response to selection of the re-sync option, skipping transcription of at least a portion of the hearing user'"'"'s voice message so that the transcription process continues on a more current segment of the hearing user'"'"'s voice message.

46. The system of claim 45 wherein, in response to selection of the re-sync option, the system skips to a pre-defined duration period that precedes the time when the re-sync selection was made so that automatically generated text is presented as fill in text for the hearing user'"'"'s voice message that corresponds to the pre-defined duration period up to a current time.

47. A system for providing voice-to-text captioning service for a hearing impaired assisted user using an assisted user'"'"'s device to communicate with a hearing user that uses a hearing user'"'"'s device, the system comprising:

a relay processor that receives voice messages generated by the hearing user during a call between the assisted user and the hearing user, the relay processor programmed to present the voice messages to a first call assistant;

a call assistant device used by the first call assistant to generate call assistant generated text corresponding to the hearing user'"'"'s voice messages;

the relay processor further programmed to;

run automated voice-to-text transcription software to generate automated text corresponding to the hearing user'"'"'s voice messages;

use the call assistant generated text, the automated text and the hearing user'"'"'s voice messages to train the voice-to-text transcription software to more accurately transcribe the hearing user'"'"'s voice messages to text, wherein the relay processor trains the voice-to-text transcription software by comparing the call assistant generated text to the automated text to identify errors in the automated text and then adjusts the software to avoid the same errors subsequently;

determine when the accuracy of the automated text exceeds an accuracy requirement threshold;

during the call and prior to the automated text exceeding the accuracy requirement threshold, transmitting the call assistant generated text to the assisted user'"'"'s device for display to the assisted user; and

subsequent to the automated text exceeding the accuracy requirement threshold, transmitting the automated text to the assisted user'"'"'s device for display to the assisted user.

48. The system of claim 47 wherein, subsequent to the automated text exceeding the accuracy requirement threshold, the relay processor stops presenting the hearing user'"'"'s voice messages to the first call assistant so that the first call assistant can transcribe a different hearing user'"'"'s voice messages.

49. The system of claim 47 further including presenting the text received at the assisted user'"'"'s device to the assisted user via a device display screen, when automated text from the relay is presented via the display screen, monitoring for an assistance request via the assisted users device and, when an assistance request is received, presenting the hearing user'"'"'s voice messages to another call assistant to again generate call assistant generated text, transmitting the call assistant generated text to the assisted user'"'"'s device for display and halting transmission of the automated text to the assisted user'"'"'s device.

50. The system of claim 49 wherein an assistance request is initiated by selecting a help button of the assisted user'"'"'s device.

51. The system of claim 47 wherein the hearing user'"'"'s voice messages received at the relay processor are received from the assisted user'"'"'s device.

52. The system of claim 47 wherein the relay processor determines when the accuracy of the automated text exceeds an accuracy requirement threshold by comparing the automated text to the call assistant generated text to generate an accuracy value for the automated text and then compares the accuracy value to the accuracy requirement.

53. The system of claim 52 wherein the call assistant generates text by revoicing the hearing user'"'"'s voice messages to voice-to-text transcription software trained to the voice of the call assistant, the call assistant generated text transmitted to the assisted user'"'"'s device and simultaneously presented to the call assistant for manual correction on a display screen, corrections by the call assistant to the call assistant generated text transmitted to the assisted user'"'"'s device for presentation in line in the text presented to the assisted user, the processor using the corrected text to train the software to the voice of the hearing user.

54. The system of claim 53 wherein, while transmitting the automated text to the assisted user'"'"'s device for display, the relay processor monitors transcription accuracy, automatically identifies accuracy degradation and, upon recognizing accuracy degradation of a threshold level, causes a re-link to a call assistant for generation of call assistant generated text to be transmitted to the assisted user'"'"'s device for display to the assisted user.

55. The system of claim 53 wherein, while transmitting the automated text to the assisted user'"'"'s device for display, the relay processor monitors transcription accuracy, automatically identifies accuracy degradation and, upon recognizing accuracy degradation of a threshold level, causes the automated text to be presented to a call assistant for correction.

56. The system of claim 55 wherein the automated text is presented to the call assistant for correction simultaneously with transmission of the automated text to the assisted user'"'"'s device for display, any corrections by the call assistant to the automated text subsequently transmitted to the assisted user'"'"'s device for in line corrections where errors are replaced by corrected text.

57. The system of claim 47 further including presenting the call assistant generated text received at the assisted user'"'"'s device to the assisted user via a device display screen and presenting an indication of a delay in text transcription on the device display screen.

58. The system of claim 57 wherein the indication of a delay includes a number of seconds indicator indicating the duration of an instantaneous delay.

59. The system of claim 57 wherein the indication of a delay includes a representation of words uttered by a hearing user that has yet to be transcribed to text at the end of a current string of displayed text wherein a length dimension of the representation of words uttered is associated with the duration of a current delay in transcription to text.

60. The system of claim 59 further including the step of presenting a re-sync option via the assisted user'"'"'s device display, detecting selection of the re-sync option and, in response to selection of the re-sync option, skipping transcription of at least a portion of the hearing user'"'"'s voice message so that the transcription process continues on a more current segment of the hearing user'"'"'s voice message.

61. The system of claim 60 wherein, in response to selection of the re-sync option, the system skips to a pre-defined duration period that precedes the time when the re-sync selection was made so that automatically generated text is presented as fill in text for the hearing user'"'"'s voice message that corresponds to the pre-defined duration period up to a current time.