Remote voice recognition

• US 9,674,351 B1
• Filed: 10/06/2016
• Issued: 06/06/2017
• Est. Priority Date: 10/06/2016
• Status: Active Grant

• Current Assignee: Sorenson Ip Holdings, LLC
• Original Assignee: Sorenson Ip Holdings, LLC
• Inventors: Mason, Joseph J.
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US15/287,370
• Time in Patent Office: 243 Days
• Field of Search: 348 1408, 348461, 348468, 348 1403, 379 52, 704200, 704235, 382218, 4554041, 4555561, 707731
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: H04M 2201/40 using speech recognition
• CPC Class Codes: H04M 2201/60 Medium conversion
• CPC Class Codes: H04M 2203/654 Pre, in or post-call message
• CPC Class Codes: H04M 3/42391 where the subscribers are hearing-impaired persons, e.g. telephone devices for the deaf
• CPC Class Codes: H04M 3/5183 Call or contact centers with computer-telephony arrangements

Abstract

According to one or more aspects of the present disclosure operations related to performing captioning may include receiving, from a first user device, first audio data. The operations may further include directing the first audio data to a remotely located call-assistant device and receiving, from the call-assistant device, second audio data that is related to the first audio data and that is derived from speech of a call assistant. The operations may also include accessing, with a captioning software application, voice profile data of the call assistant and generating caption data that includes a transcription of the second audio data. The operations may also include generating, based on the transcription, screen data related to the captioning software application, in which the screen data includes the transcription. In addition, the operations may include directing the screen data to the call-assistant device and directing the caption data to the first user device.

Claims

1. A system comprising:

a first user device of a first user, the first user device being configured to perform operations related to a captioning session;

a call-assistant device of a call assistant of a captioning system, the call-assistant device being remotely located from the first user device; and

an administrative system communicatively coupled to and remotely located from the call-assistant device and the first user device, the administrative system being configured to;

spin up a virtual computing environment based on a golden image that is a template for the virtual computing environment, the virtual computing environment being configured to run a captioning software application and being dedicated to the call assistant;

receive, from the first user device, a request to initiate a captioning session;

establish the captioning session with the first user device;

assign the captioning session to the call assistant;

receive, from the first user device and in response to establishing the captioning session, first audio data that is derived from a second user device that is performing a communication session with the first user device;

direct the first audio data to the call-assistant device in response to the administrative system being assigned to the call assistant;

receive from the call-assistant device, second audio data that is related to the first audio data and that is derived from speech of the call assistant;

access, with the captioning software application, voice profile data of the call assistant based on the captioning session being assigned to the call assistant and based on the virtual computing environment being dedicated to the call assistant;

generate, with the captioning software application, caption data that includes a transcription of the second audio data, the captioning software application being configured to use the accessed voice profile data to generate the caption data;

generate, based on the transcription, screen data related to the captioning software application, the screen data including the transcription;

direct the screen data to the call-assistant device; and

direct the caption data to the first user device.

2. The system of claim 1, wherein:

the call-assistant device is configured to;

receive, from the call assistant, a modification command related to modification of the transcription; and

direct the modification command to the administrative system; and

the administrative system is configured to;

modify, with the captioning software application, the caption data based on the modification command;

modify the screen data based on the modified caption data;

direct the modified screen data to the call-assistant device; and

direct the modified caption data to the first user device.

3. The system of claim 1, wherein the virtual computing environment includes one or more of the following:

an instance of an operating system, a virtual machine, and an instance of the captioning software application.

4. The system of claim 1, wherein the virtual computing environment is configured to:

establish the captioning session with the first user device;

receive the first audio data from the first user device;

direct the first audio data to the call-assistant device;

receive the second audio data from the call-assistant device;

generate the screen data;

direct the screen data to the call-assistant device; and

direct the caption data to the first user device.

5. The system of claim 1, wherein the call-assistant device is configured as a thin-client device.

6. The system of claim 1, wherein the operations further comprise spin up the virtual computing environment as a customized virtual computing environment for the call assistant based on the voice profile data.

7. A system comprising:

one or more processors; and

one or more non-transitory computer-readable storage media communicatively coupled to the one or more processors and configured to store instructions that, when executed by the one or more processors, cause the system to perform operations related to a captioning session, the operations comprising;

receive, from a first user device, first audio data that is derived from a second user device that is performing a communication session with the first user device, the first user device being configured to perform operations related to a captioning session;

direct the first audio data to a remotely located call-assistant device;

receive from the call-assistant device, second audio data that is related to the first audio data and that is derived from speech of a call assistant of the call-assistant device;

access, with a captioning software application running in a virtual computing environment, voice profile data of the call assistant;

spin up the virtual computing environment as a customized virtual computing environment for the call assistant based on the voice profile data and based on a golden image that is a template for the virtual computing environment;

generate, with the captioning software application, caption data that includes a transcription of the second audio data, the captioning software application being configured to use the accessed voice profile data to generate the caption data;

generate, based on the transcription, screen data related to the captioning software application, the screen data including the transcription;

direct the screen data to the call-assistant device; and

direct the caption data to the first user device.

8. The system of claim 7, wherein the operations further comprise:

receive, by the virtual computing environment from the first user device, a request to initiate the captioning session; and

establish, by the virtual computing environment, the captioning session with the first user device.

9. The system of claim 7, wherein the operations further comprise:

assign the captioning session to the call assistant; and

assign the captioning session to the customized virtual computing environment based on the captioning session being assigned to the call assistant and based on the customized virtual computing environment being customized for the call assistant.

10. The system of claim 7, wherein the operations further comprise:

receive, from the call-assistant device, a modification command related to modification of the transcription;

modify, with the captioning software application, the caption data based on the modification command;

modify the screen data based on the modified caption data;

direct the modified screen data to the call-assistant device; and

direct the modified caption data to the first user device.

11. The system of claim 7, wherein the virtual computing environment includes one or more of the following:

an instance of an operating system, a virtual machine, and an instance of the captioning software application.

12. The system of claim 7, wherein the captioning software application running on the virtual computing environment is configured to:

receive the first audio data from the first user device;

direct the first audio data to the call-assistant device;

receive the second audio data from the call-assistant device;

generate the screen data;

direct the screen data to the call-assistant device; and

direct the caption data to the first user device.

13. A method of performing captioning operations, the method being performed by a computing system and comprising:

spinning up a first virtual computing environment based on a golden image that is template for the first virtual computing environment and based on first profile data of a first call assistant of a captioning system, the first virtual computing environment being dedicated to the first call assistant and being configured to run a first instance of a captioning software application;

spinning up a second virtual computing environment based on the golden image and based on second profile data of a second call assistant of the captioning system, the second virtual computing environment being dedicated to the second call assistant and being configured to run a second instance of the captioning software application;

assigning a first captioning session to the first call assistant;

assigning the first captioning session to the first virtual computing environment based on the first virtual computing environment being dedicated to the first call assistant and based on the first captioning session being assigned to the first call assistant;

assigning a second captioning session to the second call assistant;

assigning the second captioning session to the second virtual computing environment based on the second virtual computing environment being dedicated to the second call assistant and based on the second captioning session being assigned to the second call assistant;

receiving, by the first instance of the captioning software application, first audio data from a remotely located first call-assistant device of the first call assistant, the first audio data being derived from speech of the first call assistant;

receiving, by the second instance of the captioning software application, second audio data from a remotely located second call-assistant device of the second call assistant, the second audio data being derived from speech of the second call assistant;

generating, with the first instance of the captioning software application, first caption data that includes a first transcription of the first audio data, the first instance of the captioning software application being configured to use the first profile data to generate the first caption data;

generating, with the second instance of the captioning software application, second caption data that includes a second transcription of the second audio data, the second instance of the captioning software application being configured to use the second profile data to generate the second caption data;

generating, based on the first transcription, first screen data related to the first instance of the captioning software application, the first screen data including the first transcription;

generating, based on the second transcription, second screen data related to the second instance of the captioning software application, the second screen data including the second transcription;

directing the first screen data to the first call-assistant device;

directing the second screen data to the second call-assistant device;

directing the first caption data to a first user device participating in a first communication session with a first other user device; and

directing the second caption data to a second user device participating in a second communication session with a second other user device.

14. The method of claim 13, wherein:

the first profile data includes first voice profile data of the first call assistant; and

the second profile data includes second voice profile data of the second call assistant.

15. The method of claim 14, wherein:

generating the first caption data includes accessing, by the first instance of the captioning software application, the first voice profile data and generating the first caption data based on the first voice profile data; and

generating the second caption data includes accessing, by the second instance of the captioning software application, the second voice profile data and generating the second caption data based on the second voice profile data.

16. The method of claim 13, further comprising:

receiving, from the first call-assistant device, a first modification command related to modification of the first transcription;

modifying, with the first instance of the captioning software application, the first caption data based on the first modification command;

modifying the first screen data based on the modified first caption data;

communicating the modified first screen data to the first call-assistant device;

communicating the modified first caption data to the first user device;

receiving, from the second call-assistant device, a second modification command related to modification of the second transcription;

modifying, with the second instance of the captioning software application, the second caption data based on the second modification command;

modifying the second screen data based on the modified second caption data;

communicating the modified second screen data to the second call-assistant device; and

communicating the modified second caption data to the second user device.

17. The method of claim 13, wherein the first virtual computing environment and the second virtual computing environment include one or more of the following:

an instance of an operating system, a virtual machine, and an instance of the captioning software application.

18. The method of claim 13, wherein the first call-assistant device and the second call-assistant device are configured as thin-client devices.

19. At least one non-transitory computer-readable media configured to store one or more instructions that when executed by at least one computing system perform the method of claim 13.

20. A method of performing captioning operations, the method being performed by a computing system and comprising:

spinning up a virtual computing environment as a customized virtual computing environment for a call assistant based on voice profile data of the call assistant and based on a golden image that is a template for the virtual computing environment;

receiving, from a first user device, first audio data that is derived from a second user device participating in a communication session with the first user device, the first user device being configured to perform operations related to a captioning session;

directing the first audio data to a remotely located call-assistant device;

receiving from the call-assistant device, second audio data that is related to the first audio data and that is derived from speech of the call assistant;

accessing, with a captioning software application running in the virtual computing environment, the voice profile data of the call assistant;

generating, with the captioning software application, caption data that includes a transcription of the second audio data, the captioning software application being configured to use the accessed voice profile data to generate the caption data; and

directing the caption data to the first user device.

21. The method of claim 20, further comprising assigning the captioning session to the customized virtual computing environment based on the captioning session being assigned to the call assistant and based on the customized virtual computing environment being customized for the call assistant.

22. The method of claim 20, further comprising:

receiving, by the virtual computing environment from the first user device, a request to initiate the captioning session; and

establishing, by the virtual computing environment, the captioning session with the first user device.

23. The method of claim 20, wherein the virtual computing environment includes one or more of the following:

an instance of an operating system, a virtual machine, and an instance of the captioning software application.