System and method for captioning media

• US 9,696,881 B2
• Filed: 08/19/2013
• Issued: 07/04/2017
• Est. Priority Date: 01/15/2013
• Status: Active Grant

• Current Assignee: ViKi Inc. (Rakuten Group, Inc.)
• Original Assignee: ViKi Inc. (Rakuten Group, Inc.)
• Inventors: Pornprasitsakul, Nuttanart, Lee, Sunjin
• Primary Examiner(s): Chow, Doon
• Assistant Examiner(s): Chiusano, Andrew T
• Application Number:
  US13/970,034
• Publication Number:
  US 20140201631A1
• Time in Patent Office: 1,415 Days
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel

Abstract

A method for collaboratively captioning streamed media, the method including: rendering a visual representation of the audio at a first device, receiving segment parameters for a first media segment from the first device, rendering the visual representation of the audio at a second device, the second device different from the first device, and receiving a caption for the first media segment from the second device.

Claims

1. A method for captioning streamed media having video and audio, the method comprising:

streaming the media to a first device associated with a first user account from a remote media repository;

rendering a visual representation of the audio along a timeline of the media at the first device;

receiving segment parameters that define a media segment bounded by a first timestamp and a second timestamp from the first user account;

generating a caption file comprising a caption associated with the first timestamp and the second timestamp;

receiving a first annotation associated with the defined media segment from the first user account;

editing the caption file, comprising saving the received first annotation as a version of the annotation for the caption associated with the first timestamp and the second timestamp;

streaming the media to a second device associated with a second user account from the remote media repository;

rendering the visual representation of the audio with icons corresponding to the first timestamp and second timestamp of the defined media segment;

receiving a second annotation associated with the defined media segment from the second user account;

editing the caption file, comprising saving the received second annotation as a version of the annotation for the caption associated with the first timestamp and the second timestamp;

automatically selecting one of the first or second annotations to be presented as the current version of the caption, wherein automatically selecting one of the first or second annotations comprises comparing the first and second annotations to an automatic machine translation of the audio of the defined media segment and selecting the annotation most similar to the machine translation;

streaming the media to a third device associated with a third user account; and

rendering the selected annotation for the defined media segment over video frames between the first timestamp and the second timestamp at the third device.

2. The method of claim 1, wherein receiving a first or a second annotation comprises:

receiving a selection of the defined media segment; and

receiving the first or the second annotation at a caption input field associated with the selected media segment.

3. The method of claim 2, wherein the caption input field comprises a text input associated with a video frame within the selected media segment.

4. The method of claim 1, further comprising receiving a boundary change from the second user account and updating the caption in the caption file with a timestamp corresponding to the changed boundary.

5. The method of claim 1, further comprising receiving segment parameters that define a second media segment bounded by a third and a fourth timestamp from the second user account;

and adding the third and fourth timestamp associated with a new caption to the caption file.

6. The method of claim 1, wherein receiving segment parameters that define the media segment comprises receiving a selection of a segment of the visual representation.

7. The method of claim 6, wherein receiving a selection of a segment of the visual representation comprises:

receiving an input at the first timestamp;

playing the video and audio, starting from the first timestamp; and

detecting removal of the input and setting the second timestamp to the timestamp at which the input was removed, wherein the input is sustained during video and audio play.

8. The method of claim 7, wherein the input is a cursor selection of a position on the visual representation of the audio.

9. The method of claim 1, wherein the caption file is stored at the remote media repository.

10. The method of claim 1, wherein

selecting the annotation most similar to the machine translation comprises selecting one or more annotations having a similarity to the machine translation beyond a predetermined threshold or removing one or more annotations having a difference from the machine translation beyond a predetermined threshold,

wherein, in response to selecting more than one annotation having a similarity to the machine translation beyond a predetermined threshold, iteratively or randomly selecting and presenting the more than one annotation.

11. The method of claim 10, wherein having a similarity to the machine translation is determined by phrase convergence, similarity of words, or similarity of grammar and having a difference from the machine translation is determined by phrase divergence, difference of words, or difference of grammar.

12. A method for collaboratively captioning media including audio, the method comprising:

rendering a visual representation of the audio at a first device associated with a first user account;

receiving segment parameters corresponding to boundaries of a first media segment from the first user account;

receiving a first annotation for the first media segment from the first user account;

rendering the visual representation of the audio at a second device associated with a second user account, the second user account different from the first user account, the visual representation further comprising icons corresponding to the boundaries of the first media segment;

receiving a second annotation for the first media segment from the second user account;

automatically selecting one of the first or second annotations to be presented with the first media segment, wherein automatically selecting one of the first or second annotations comprises comparing the first and second annotations to an automatic machine translation of the audio of the first media segment and selecting the annotation most similar to the machine translation;

rendering the visual representation of the audio at a third device associated with a third user account, the third user account different from the first and second user accounts, the visual representation further comprising icons corresponding to the boundaries of the first media segment; and

rendering the selected annotation for the first media segment over video frames occurring between the boundaries of the first media segment.

13. The method of claim 12, further comprising receiving segment parameters corresponding to boundaries of a second media segment from the second user account.

14. The method of claim 12, further comprising receiving segment parameters corresponding to boundaries of a second media segment from the third user account.

15. The method of claim 12, further comprising receiving a segment parameter change for the first media segment from the second device and updating the first media segment boundaries based on the segment parameter change.

16. The method of claim 12, wherein the segment parameters are indicative of a first timestamp and a second timestamp, wherein receiving segment parameters corresponding to boundaries of the first media segment comprises progressing the media along a timeline;

receiving a first input at a first timestamp; and

receiving a second input at the second timestamp.

17. The method of claim 12, further comprising receiving segment parameters corresponding to successive media segments from the first device.

18. The method of claim 17, wherein receiving segment parameters corresponding to successive media segments from the first device comprises progressing the media along a timeline;

during sustained receipt of a first input;

receiving a second input at a first timestamp;

receiving a third input at a second timestamp; and

receiving a fourth input at a third timestamp;

wherein the first media segment is defined between the first and second timestamps, and a second media segment is defined between the second and third timestamps.

19. The method of claim 12, further comprising streaming the media to the first, second, and third devices from a remote media repository.

20. The method of claim 12, wherein

selecting the annotation most similar to the machine translation comprises selecting one or more annotations having a similarity to the machine translation beyond a predetermined threshold or removing one or more annotations having a difference from the machine translation beyond a predetermined threshold,

wherein, in response to selecting more than one annotation having a similarity to the machine translation beyond a predetermined threshold, iteratively or randomly selecting and presenting the more than one annotation.

21. The method of claim 20, wherein having a similarity to the machine translation is determined by phrase convergence, similarity of words, or similarity of grammar and having a difference from the machine translation is determined by phrase divergence, difference of words, or difference of grammar.