System for generating captions for live video broadcasts

• US 9,026,446 B2
• Filed: 06/10/2011
• Issued: 05/05/2015
• Est. Priority Date: 06/10/2011
• Status: Active Grant

• Current Assignee: Deluxe 3D LLC (Deluxe Corp.), Deluxe Digital Distribution Inc. (Deluxe Corp.), Deluxe Digital Studios, Inc. (Deluxe Corp.), Deluxe Entertainment Services Inc., Deluxe Laboratories LLC, Deluxe Media Inc., Deluxe One, LLC, Softitler Net, Inc., Deluxe Creative Services Inc. (The Framestore Ltd.)
• Original Assignee: Morgan Fiumi
• Inventors: Fiumi, Morgan
• Primary Examiner(s): Shah, Paras D
• Assistant Examiner(s): Sirjani, Fariba
• Application Number:
  US13/158,331
• Publication Number:
  US 20120316882A1
• Time in Patent Office: 1,425 Days
• Field of Search: 704/240
• US Class Current: 704/270
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

An adaptive workflow system can be used to implement captioning projects, such as projects for creating captions or subtitles for live and non-live broadcasts. Workers can repeat words spoken during a broadcast program or other program into a voice recognition system, which outputs text that may be used as captions or subtitles. The process of workers repeating these words to create such text can be referred to as respeaking. Respeaking can be used as an effective alternative to more expensive and hard-to-find stenographers for generating captions and subtitles.

Claims

1. A method of performing distributed caption generation, the method comprising:

programmatically selecting first and second respeakers to perform respeaking with a voice recognition engine for a broadcast program based at least in part on past performance ratings of the first and second respeakers;

receiving first text generated by the first respeaker for inclusion in the broadcast program;

receiving second text generated by the second respeaker for inclusion in the broadcast program, wherein the second text is being received as backup in case receipt of the first text is interrupted;

electronically outputting the first text for inclusion as captions in the broadcast program;

determining whether receipt of the first text is interrupted;

in response to determining that receipt of the first text is interrupted, outputting the second text for inclusion in the broadcast program; and

programmatically calculating new performance ratings for the first and second respeakers, the new performance ratings configured to be used to assign the first or second respeaker to a subsequent broadcast program, said calculating the new performance ratings for the first and second respeakers comprising taking into account at least the following factors;

network connections of the first and second respeakers;

whether the first and second respeakers train the voice recognition engine, subsequent to the broadcast program, with words incorrectly transcribed during the broadcast program; and

whether the first and second respeakers train the voice recognition engine with new words not previously recognized by the voice recognition engine;

wherein the new performance ratings are negatively impacted if the first and second respeakers fail to train the voice recognition engine and are positively impacted if the first and second respeakers train the voice recognition engine; and

wherein at least said determining and said calculating are implemented by a computer system comprising computer hardware.

2. The method of claim 1, wherein said determining that receipt of the first text is interrupted comprises pinging a computer system operated by the first respeaker.

3. The method of claim 2, wherein a timeout in said pinging results in the determination that the first text is interrupted.

4. The method of claim 1, wherein said calculating the new performance ratings further comprises evaluating one or more of the following performance factors:

accuracy, timeliness, availability, rate, and professionalism.

5. The method of claim 1, wherein said broadcast program is broadcast over the air, via cable, via satellite, and/or via a computer network.

6. The method of claim 1, wherein said past performance ratings are based in part on an accuracy rating with respect to voice recognition performed by a voice recognition system for the first respeaker and the second respeaker.

7. A method of performing distributed caption generation, the method comprising:

selecting a respeaker to perform respeaking with a voice recognition engine for a broadcast program based at least in part on a past performance rating of the respeaker;

receiving text generated by the respeaker for inclusion in the broadcast program;

electronically outputting the text for inclusion in the broadcast program;

calculating a new performance rating for the respeaker, the new performance rating configured to be used to evaluate whether to assign the respeaker to a subsequent broadcast program, said calculating the new performance rating comprising taking into account at least the following factors;

a network connection of the respeaker;

whether the respeaker trains the voice recognition engine, subsequent to the broadcast program, with words incorrectly transcribed during the broadcast program;

whether the respeaker trains the voice recognition engine with new words not previously recognized by the voice recognition engine; and

wherein the new performance rating is negatively impacted if the respeaker fails to train the voice recognition engine and is positively impacted if the respeaker trains the voice recognition engine; and

wherein at least said calculating is implemented by a computer system comprising computer hardware.

8. The method of claim 7, wherein said calculating the new performance ratings further comprises evaluating one or more of the following performance factors:

accuracy, timeliness, availability, rate, and professionalism.

9. The method of claim 7, wherein said calculating the new performance rating further comprises evaluating whether a computing system of the respeaker reliably transmits the text during the broadcast program.

10. A system for performing distributed caption generation, the system comprising:

a project management module configured to select a respeaker to perform respeaking with a voice recognition engine for a broadcast based at least in part on a past performance rating of the respeaker;

a project network application comprising a respeaking module configured to;

provide functionality for the respeaker to generate text responsive to audio from a broadcast,

receive text generated by the respeaker, and

electronically output the text for inclusion in the broadcast; and

a worker ratings calculator comprising computer hardware, the worker ratings calculator configured to calculate a new performance rating for the respeaker, the new performance rating configured to be used to evaluate whether to assign the respeaker to a subsequent broadcast program, wherein the worker rating calculator is further configured to calculate the new performance rating by taking into account at least the following factors;

a network connection of the respeaker;

whether the respeaker trains the voice recognition engine, subsequent to the broadcast program, with words incorrectly transcribed during the broadcast program; and

whether the respeaker trains the voice recognition engine with new words not previously recognized by the voice recognition engine;

wherein the new performance rating is negatively impacted if the respeaker fails to train the voice recognition engine and is positively impacted if the respeaker trains the voice recognition engine.

11. The system of claim 10, wherein the worker ratings calculator is further configured to calculate the new performance ratings by at least evaluating one or more of the following performance factors:

accuracy, timeliness, availability, rate, and professionalism.

12. Non-transitory physical computer storage comprising instructions stored thereon for implementing, in one or more processors, operations for performing distributed caption generation, the operations comprising:

selecting a respeaker to perform respeaking with a voice recognition engine for a first program based at least in part on a past performance rating of the respeaker;

receiving text generated by the respeaker for inclusion in the first program;

electronically outputting the text for inclusion in the first program; and

programmatically calculating a new performance rating for the respeaker, the new performance rating configured to be used to evaluate whether to assign the respeaker to a subsequent program, wherein said calculating the new performance rating comprises taking into account at least the following factors;

a network connection of the respeaker;

whether the respeaker trains the voice recognition engine, subsequent to the first program, with words incorrectly transcribed during the broadcast program; and

whether the respeaker trains the voice recognition engine with new words not previously recognized by the voice recognition engine;

wherein the new performance rating is negatively impacted if the respeaker fails to train the voice recognition engine and is positively impacted if the respeaker trains the voice recognition engine.

13. The non-transitory physical computer storage of claim 12, wherein said calculating the new performance ratings comprises evaluating one or more of the following performance factors:

accuracy, timeliness, availability, rate, and professionalism.

14. The non-transitory physical computer storage of claim 12, wherein the first program comprises a live broadcast.

15. The non-transitory physical computer storage of claim 12, wherein the first program comprises a video.

16. The non-transitory physical computer storage of claim 12, wherein said outputting the text comprises supplying the text as subtitles for the first program.

17. The non-transitory physical computer storage of claim 12, wherein said outputting the text comprises supplying the text as captions for the first program.

18. The non-transitory physical computer storage of claim 12, in combination with a computer system comprising computer hardware.