Synchronization mechanism for multimedia captioning and audio description

• US 7,631,326 B2
• Filed: 12/26/2002
• Issued: 12/08/2009
• Est. Priority Date: 12/26/2002
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kaneko, Masahiko
• Primary Examiner(s): Pendleton; Brian T
• Assistant Examiner(s): Saint Cyr; Jean D
• Application Number:
  US10/330,422
• Publication Number:
  US 20040128702A1
• Time in Patent Office: 2,539 Days
• Field of Search: 725/25, 725/112, 725/136, 725/32, 725/42, 725/137, 348/62
• US Class Current: 725/32
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 21/854 Content authoring
• CPC Class Codes: H04N 7/08 Systems for the simultaneous or sequential transmission of more than one television signal, e.g. additional information signals, the signals occupying wholly or partially the same frequency band , e.g. by time division H04N7/007 takes precedence
• CPC Class Codes: H04N 7/0885 for the transmission of subtitles
• CPC Class Codes: H04N 7/165 Centralised control of user terminal subsequent to an upstream request signal H04N7/17345; Registering at central by two-way working H04N7/17309

Abstract

A method for outputting supplemental media streams with a main media stream is presented. The method monitors for triggering events and, upon detecting a triggering event, causes one or more supplemental media streams associated with the triggering event to output. The associated supplemental media streams may be output either synchronously or asynchronously with the main media stream. Synchronous supplemental media streams are output concurrently with the main media stream. Asynchronously output supplemental media streams suspend the main media stream while they are output and terminate any currently outputting supplemental media streams. A supplemental media stream may cause a conditional triggering event to trigger a second supplemental media stream to output synchronously with the first. A second main media stream with associated supplemental media streams may be output asynchronously as a compiled supplemental media stream in connection with a first main media stream.

Claims

1. A method for outputting one or more supplemental media streams designed to assist sense impaired people in conjunction with a main media stream, comprising:

outputting a main media stream on a display device;

repeatedly monitoring for and detecting triggering events while outputting the main media stream; and

upon detecting a triggering event;

determining which of the one or more supplemental media streams are associated with the triggering event;

determining for each associated supplemental media stream whether each associated supplemental media stream is to be output synchronously or asynchronously with the main media stream, wherein the determination whether to output each associated supplemental media stream synchronously or asynchronously is made automatically without user intervention based on information provided in either the triggering event or each supplemental media stream; and

iteratively outputting each associated supplemental media stream either synchronously or asynchronously with the main media stream according to the results of the previous determinations such that upon determining that a supplemental media stream is to be output asynchronously with the main media stream, the output of the main media stream is automatically suspended without user intervention while the supplemental media stream is output on the display device.

2. The method of claim 1, further comprising determining a plurality of supplemental media streams associated with the triggering event and wherein the plurality of associated supplemental media streams are iteratively output according to a predetermined order among the associated supplemental media streams.

3. The method of claim 2, wherein the predetermined order among associated supplemental media streams is established according to priority values associated with each supplemental media stream.

4. The method of claim 1, further comprising terminating a second supplemental media stream currently being output when the associated supplemental media stream is output asynchronously.

5. The method of claim 1, further comprising resuming output of the main media stream after asynchronously outputting the associated supplemental media stream.

6. The method of claim 5, wherein the triggering events occur according to the timing of the main media stream.

7. The method of claim 6, wherein the timing of the main media stream is suspended when the main media stream is suspended.

8. The method of claim 1, wherein a first associated supplemental media stream iteratively causes a second associated supplemental media stream to output when the first associated supplemental media stream terminates its output.

9. The method of claim 1, wherein an associated supplemental media stream issues a conditional triggering event that causes a conditional supplemental media stream to output synchronously with the associated supplemental media stream.

10. The method of claim 9, wherein the associated supplemental media stream selectively issues the conditional triggering event according to a condition external to the associated supplemental media stream.

11. The method of claim 1, wherein determining which of the one or more supplemental media streams are associated with the triggering event comprises selectively excluding, according to an external condition, a supplemental media stream that would otherwise be associated with the triggering event.

12. The method of claim 11, wherein the external condition identifies the type of supplemental media stream to be selectively excluded.

13. The method of claim 1, wherein a supplemental media stream comprises either an audio description stream, a visual description stream, or a compiled media stream.

14. The method of claim 13, wherein a compiled media stream comprises a compiled main media stream and one or more compiled supplemental media streams associated with the compiled media stream.

15. The method of claim 14, wherein outputting a compiled media stream comprises:

repeatedly monitoring for and detecting compiled triggering events while outputting the compiled main media stream; and

upon detecting a compiled triggering event;

determining which of the one or more compiled supplemental media streams are associated with the compiled triggering event;

determining for each associated compiled supplemental media stream whether each associated compiled supplemental media stream is to be output synchronously or asynchronously with the compiled main media stream; and

iteratively outputting each associated compiled supplemental media stream either synchronously or asynchronously with the compiled main media stream in accordance with the results of the previous determinations such that upon determining that a compiled supplemental media stream is to be output asynchronously with the main media stream, the output of the main media stream is automatically suspended without user intervention while the compiled supplemental media stream is output.

16. The method of claim 15, wherein a compiled supplemental media stream comprises either an audio description stream, a visual description stream, or a compiled media stream.

17. The method of claim 15, wherein the compiled triggering events occur according to the timing of the compiled main media stream.

31. The method of claim 1 wherein the main media stream is a stream comprising video data and audio data and wherein the supplemental media streams comprise closed-captioning text, and wherein automatically suspending the main media stream comprises pausing the video and audio of the main media stream such that the closed-captioning text of the supplemental data stream to be output asynchronously is displayed together with the paused video data of the main media stream.

18. A computer readable storage medium having computer executable instructions for carrying out the method comprising the steps of:

outputting a main media stream;

repeatedly monitoring for and detecting triggering events while outputting the main media stream; and

upon detecting a triggering event;

determining which of one or more supplemental media streams are associated with the triggering event;

determining for each associated supplemental media stream whether each associated supplemental media stream is to be output synchronously or asynchronously with the main media stream, wherein the determination whether to output each associated supplemental media stream synchronously or asynchronously is made automatically without user intervention based on information provided in either the triggering event or each supplemental media stream; and

iteratively outputting each associated supplemental media stream either synchronously or asynchronously with the main media stream according to the results of the previous determinations to a display device for display on said device such that upon determining that a supplemental media stream is to be output asynchronously with the main media stream, the output of the main media stream is automatically suspended without user intervention while the supplemental media stream is output.

19. The computer readable storage medium of claim 18, wherein a supplemental media stream comprises either an audio description stream, a visual description stream, or a compiled media stream, a compiled media stream comprising a compiled main media stream and one or more compiled supplemental media streams associated with the compiled media stream, and outputting a compiled media stream comprises:

repeatedly monitoring for and detecting compiled triggering events while outputting the compiled main media stream; and

upon detecting a compiled triggering event;

determining which of the one or more compiled supplemental media streams are associated with the compiled triggering event;

determining for each associated compiled supplemental media stream whether each associated compiled supplemental media stream is to be output synchronously or asynchronously with the compiled main media stream; and

iteratively outputting each associated compiled supplemental media stream either synchronously or asynchronously with the compiled main media stream in accordance with the results of the previous determinations such that upon determining that a compiled supplemental media stream is to be output asynchronously with the main media stream, the output of the main media stream is automatically suspended without user intervention while the compiled supplemental media stream is output.

20. A system for outputting at least one supplemental media stream designed to assist sense impaired people in conjunction with a main media stream, the system comprising:

a media stream output device for outputting a main media stream and at least one supplemental media stream, wherein the at least one supplemental media stream comprises captioning that displays in textual format either dialog of characters that are being displayed in the main media stream or sounds of events that occur in the main media stream; and

a media stream output controller communicatively coupled to the media stream output device for;

receiving a main media stream and at least one supplemental media stream;

outputting the main media stream via the media stream output device;

monitoring for and detecting triggering events associated with the output of the main media stream; and

upon detecting a triggering event;

determining which of the at least one supplemental media streams are associated with the detected triggering event;

determining for each associated supplemental media stream whether each associated supplemental media stream is to be output synchronously or asynchronously with the main media stream, wherein the determination whether to output each associated supplemental media stream synchronously or asynchronously is made automatically without user intervention based on information provided in either the triggering event or each supplemental media stream; and

iteratively outputting each associated supplemental media stream, via the media stream output device, either synchronously or asynchronously with the main media stream in accordance with the results of the previous determinations such that upon determining that a supplemental media stream is to be output asynchronously with the main media stream, the output of the main media stream is automatically suspended without user-intervention while the supplemental media stream is output.

21. The system of claim 20, wherein the associated supplemental media streams are iteratively output according to a predetermined order among the associated supplemental media streams.

22. The system of claim 21, wherein the predetermined order among the associated supplemental media streams is established according to priority values assigned to each supplemental media stream.

23. The system of claim 20, wherein the media stream output controller is further for terminating a second supplemental media stream currently being output when the associated supplemental media stream is output asynchronously.

24. The system of claim 20, wherein the media stream output controller is further for resuming output of the main media stream after asynchronously outputting the associated supplemental media stream.

25. The system of claim 20, wherein a first associated supplemental media stream iteratively causes a second associated supplemental media stream to output when the first associated supplemental media stream terminates its output.

26. The system of claim 20, wherein an associated supplemental media stream issues a conditional triggering event that causes a conditional supplemental media stream to output synchronously with the associated supplemental media stream.

27. The system of claim 26, wherein the associated supplemental media stream selectively issues the conditional triggering event according to a condition external to the associated supplemental media stream.

28. The system of claim 20, wherein a supplemental media stream comprises an audio description stream, a visual description stream, or a compiled media stream.

29. The system of claim 28, wherein a compiled media stream comprises a compiled main media stream and at least one compiled supplemental media stream associated with the compiled media stream.

30. A media display system for outputting one or more supplemental media streams designed to assist sense impaired people in conjunction with outputting a main media stream, the system comprising:

a monitoring means for monitoring and detecting triggering events associated while outputting the main media stream;

a supplemental media stream selection means for selecting which of the one or more supplemental media streams are associated with a detected triggering event;

a determination means for determining for each associated supplemental media stream whether each associated supplemental media stream is to be output synchronously or asynchronously with the main media stream, wherein the determination whether to output each associated supplemental media stream synchronously or asynchronously is made automatically without user intervention based on information provided in either the triggering event or each supplemental media stream;

a control means for iteratively outputting each associated supplemental media stream either synchronously or asynchronously with the main media stream according to the results of the determinations by the determination means, terminating any currently outputting supplemental media streams when an associated supplemental media stream is output asynchronously, and suspending the main media stream while the associated supplemental media stream is output asynchronously such that upon the control means determining that a supplemental media stream is to be output asynchronously with the main media stream, the output of the main media stream is automatically suspended without user intervention while the supplemental media stream is output; and

a display means for displaying the media streams.