Synchronization mechanism and the implementation for multimedia captioning and audio descriptions

• US 7,676,737 B2
• Filed: 04/10/2003
• Issued: 03/09/2010
• Est. Priority Date: 04/10/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kaneko, Masahiko
• Primary Examiner(s): Ries; Laurie
• Assistant Examiner(s): NGUYEN, CHAU T
• Application Number:
  US10/412,805
• Publication Number:
  US 20040205830A1
• Time in Patent Office: 2,525 Days
• Field of Search: 715201-203, 725/9, 725/37, 725/40, 725/32, 725/137
• US Class Current: 715/203
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04N 21/4143 embedded in a Personal Computer [PC]
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 21/8455 involving pointers to the content, e.g. pointers to the I-frames of the video stream
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/858 Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
• CPC Class Codes: H04N 5/073 for mutually locking plural sources of synchronising signals, e.g. studios or relay stations

Abstract

A system for outputting a media stream and supplemental media streams is provided. A media stream manager outputs the media stream and associated supplemental media streams on a media stream output device. A timing module generates triggering events in conjunction with the output of the media streams according to event data. The media stream manager monitors for and detects the triggering events during the output of the media streams. Upon detecting a triggering event, the media stream manager determines which supplemental media streams are associated with the triggering event and iteratively outputs each associated supplemental media stream to the media stream output device. Supplemental media streams are output either synchronously or asynchronously according to previously determined information. A supplemental media stream that is output asynchronously to the media stream suspends the media stream and the timing of the media stream while the supplemental media stream is output.

Claims

1. A system for outputting at least one supplemental media stream designed to assist sense-impaired people in conjunction with a main media stream, the system comprising:

(a) a media stream output device that outputs a main media stream and a supplemental media stream, wherein the supplemental media stream comprises captioning that displays in textual format either dialog of characters that are being displayed in the main media stream or sounds of events that occur in the main media stream;

(b) a timing module that monitors the timing of the output of the main media stream and generates triggering events associated with the timing of the output of the main media stream; and

(c) a media stream manager communicatively coupled to the media stream output device and to the timing module, and that;

(i) obtains the main media stream;

(ii) outputs the main media stream via the media stream output device;

(iii) monitors for and detects triggering events associated with the timing of the output of the main media stream generated by the timing module; and

(iv) upon detecting a triggering event;

(A) determines that the supplemental media stream is associated with the triggering event

and determines whether the supplemental media stream includes information indicating that the supplemental media stream is to be output synchronously or asynchronously

and upon determining that the information indicates that the supplemental media stream is to be output asynchronously, outputs the supplemental media stream via the media stream output device such that the output of the main media stream is automatically suspended without user intervention while the supplemental media stream is output.

2. The system of claim 1, wherein the media stream manager initializes the timing module in conjunction with the output of the main media stream.

3. The system of claim 2, wherein the media stream manager initializes the timing module with an event table, the event table comprising a plurality of event entries, wherein each event entry identifies an associated supplemental media stream and a time during the output of the main media stream when the timing module is to generate a triggering event indicating the associated supplemental media stream is to be output.

4. The system of claim 1, wherein the media stream output device outputs a plurality of supplemental media streams, and wherein the media stream manager, upon detecting a triggering event, determines which of a plurality of supplemental media streams are associated supplemental media streams associated with the detected triggering event, and iteratively outputs the associated supplemental media streams via the media stream output device.

5. The system of claim 4, wherein iteratively outputting each associated supplemental media stream via the media stream output device comprises:

determining an order among the associated supplemental media streams; and

iteratively outputting each associated supplemental media stream via the media stream output device according to the determined order.

6. The system of claim 5, wherein determining an order among the associated supplemental media streams comprises:

obtaining a priority value associated with each associated supplemental media stream; and

ordering the associated supplemental media streams according to their associated priority values, the resultant order being the determined order.

7. The system of claim 4, wherein iteratively outputting the associated supplemental media streams via the media stream output device comprises:

determining for each associated supplemental media stream whether the associated supplemental media stream is to be output synchronously or asynchronously to the main media stream; and

iteratively outputting each associated supplemental media stream via the media stream output device either synchronously or asynchronously with the main media stream according to the results of the determinations.

8. The system of claim 7, wherein iteratively outputting each associated supplemental media stream either synchronously or asynchronously with the main media stream comprises suspending the output of the main media stream while an associated supplemental media stream is output asynchronously.

9. The system of claim 8, wherein iteratively outputting each associated supplemental media stream either synchronously or asynchronously with the main media stream further comprises causing the timing module to suspend the timing of the main media stream when the main media stream is suspended.

10. The system of claim 9, wherein iteratively outputting each associated supplemental media stream either synchronously or asynchronously with the main media stream further comprises terminating the output of any associated supplemental media stream being output when an associated supplemental media stream is output asynchronously.

11. The system of claim 10, wherein iteratively outputting each associated supplemental media stream either synchronously or asynchronously with the main media stream further comprises resuming output of the main media stream when the asynchronously output associated supplemental media stream terminates.

12. The system of claim 11, wherein iteratively outputting each associated supplemental media stream either synchronously or asynchronously with the main media stream further comprises causing the timing module to resume the timing of the main media stream when the asynchronously output associated supplemental media stream terminates.

13. The system of claim 8, wherein the timing module further monitors the timing of each associated supplemental media stream being output.

14. The system of claim 13, wherein the media stream manager initializes the timing module in conjunction with the output of each associated supplemental media stream.

15. The system of claim 8, wherein the supplemental media stream is an audio, visual, or compiled supplemental media stream.

16. The system of claim 15, wherein the compiled media stream comprises a compiled main media stream and a compiled supplemental media stream associated with the compiled main media stream, and wherein outputting the compiled media stream comprises:

asynchronously outputting the compiled main media stream to the main media stream;

monitoring for and detecting compiled triggering events while outputting the compiled main media stream; and

upon detecting a compiled triggering event, determining that the compiled supplemental media stream is associated with the compiled triggering event

and determining whether the compiled supplemental media stream is to be output synchronously or asynchronously with the compiled main media stream; and

outputting the compiled supplemental media stream either synchronously or asynchronously with the compiled main media stream according to the results of the determination.

17. The system of claim 1, wherein the media stream manager outputs the main media stream via the media stream output device comprises:

instantiating an output module for outputting the main media stream via the media stream output device according to the type of the main media stream;

binding the main media stream to the output module; and

causing the output module to output the main media stream via the media stream output device.

18. The system of claim 17, wherein the media stream manager outputs the supplemental media stream via the media stream output device comprises:

instantiating an output module for outputting the supplemental media stream via the media stream output device according to the type of the supplemental media stream;

binding the supplemental media stream to the output module; and

causing the output module to output the supplemental media stream via the media stream output device.

19. The system of claim 18, wherein the output module is a Windows Media Viewer module.

20. The system of claim 18, wherein the output module is an audiovisual viewer.

21. The system of claim 18, wherein the output module is a block viewer.

22. The system of claim 18, wherein the output module is a scrolling line viewer.

23. The system of claim 18, wherein the output module is an audio output player.

24. A method for outputting at least one supplemental media stream designed to assist sense impaired people in conjunction with a main media stream, the method comprising:

(a) obtaining a main media stream and a supplemental media stream for output on a media stream output device, wherein the supplemental media stream comprises captioning that displays in textual format either dialog of characters that are being displayed in the main media stream or sounds of events that occur in the main media stream;

(b) initializing a timing module to perform timing of the output of the main media stream and to generate triggering events associated with the timing of the output of the main media stream;

(c) outputting the main media stream via the media stream output device;

(d) monitoring for and detecting triggering events associated with the timing of the output of the main media stream generated by the timing module; and

(e) upon detecting a triggering event;

(i) determining that the supplemental media stream is associated with the detected triggering event

and determining whether the supplemental media stream includes information indicating that the supplemental media stream is to be output synchronously or asynchronously

and upon determining that the information indicates that the supplemental media stream is to be output asynchronously, outputting the supplemental media stream via the media stream output device such that the output of the main media stream is automatically suspended without user intervention while the supplemental media stream is output.

25. The method of claim 24, wherein initializing the timing module to perform timing of the output of the main media stream and generate triggering events associated with the timing of the output of the main media stream comprises initializing the timing module with an event table, the event table comprising a plurality of event entries, and wherein each event entry identifies an associated supplemental media stream and a time during the output of the main media stream when the timing module is to generate a triggering event indicating the associated supplemental media stream is to be output.

26. The method of claim 25 further comprising, upon detecting a triggering event, determining which of a plurality of supplemental media streams are associated supplemental media streams associated with the detected triggering event, and iteratively outputting the associated supplement media streams via the media stream output device.

27. The method of claim 26, wherein iteratively outputting the associated supplemental media stream via the media stream output device further comprises:

determining an order among the associated supplemental media streams; and

iteratively outputting the associated supplemental media streams via the media stream output device according to the determined order.

28. The method of claim 27, wherein determining an order among the associated supplemental media streams further comprises:

obtaining a priority value associated with each associated supplemental media stream; and

ordering the associated supplemental media streams according to their associated priority values, the resultant order being the determined order.

29. The method of claim 26, wherein iteratively outputting the associated supplemental media streams via the media stream output device further comprises:

determining for each associated supplemental media stream whether the associated supplemental media stream is to be output synchronously or asynchronously to the main media stream; and

iteratively outputting each associated supplemental media stream via the media stream output device either synchronously or asynchronously with the main media stream according to the results of the determinations.

30. The method of claim 29, wherein iteratively outputting the associated supplemental media streams either synchronously or asynchronously with the main media stream further comprises suspending the output of the main media stream while an associated supplemental media stream is output asynchronously.

31. The method of claim 30 further comprising causing the timing module to suspend timing of the output of the main media stream when the output of the main media stream is suspended.

32. The method of claim 31 further comprising terminating the output of a supplemental media stream being output when an associated supplemental media stream is output asynchronously.

33. The method of claim 32 further comprising resuming the output of the main media stream when the asynchronously output associated supplemental media stream terminates.

34. The method of claim 33 further comprising causing the timing module to resume the timing of the output of the main media stream when the output of the main media stream resumes.

35. The method of claim 30 further comprising initializing the timing module to perform timing of the output of each associated supplemental media stream and to generate triggering events associated with the timing of the output of each associated supplemental media stream.

36. The method of claim 30, wherein the supplemental media stream is an audio, visual, or compiled supplemental media stream.

37. The method of claim 36, wherein the compiled media stream comprises a compiled main media stream and a compiled supplemental media stream associated with the compiled main media stream, and wherein outputting the compiled media stream comprises:

asynchronously outputting the compiled main media stream to the main media stream;

monitoring for and detecting compiled triggering events while outputting the compiled main media stream; and

upon detecting a compiled triggering event;

determining that the compiled supplemental media stream is associated with the compiled triggering event,

and determining whether the compiled supplemental media stream is to be output synchronously or asynchronously with the compiled main media stream; and

outputting the compiled supplemental media stream either synchronously or asynchronously with the compiled main media stream according to the results of the determination.

38. The method of claim 24, wherein initializing the output of the main media stream via the media stream output device comprises:

instantiating an output module for outputting the main media stream via the media stream output device according to the type of the main media stream;

binding the main media stream to the output module; and

causing the output module to output the main media stream via the media stream output device.

39. The method of claim 38, wherein outputting the supplemental media stream via the media stream output device comprises:

instantiating an output module for outputting the supplemental media stream via the media stream output device according to the type of the supplemental media stream;

binding the supplemental media stream to the output module; and

causing the output module to output the supplemental media stream via the media stream output device.

40. The method of claim 39, wherein the output module is a Windows Media Viewer module.

41. The method of claim 39, wherein the output module is an audiovisual viewer.

42. The method of claim 39, wherein the output module is a block viewer.

43. The method of claim 39, wherein the output module is a scrolling line viewer.

44. The method of claim 39, wherein the output module is an audio output player.

45. A computer-readable medium having computer-executable instructions for carrying out a method comprising the steps of:

(a) obtaining a main media stream and a supplemental media stream for output on a media stream output devices, wherein the supple mental media stream comprises captioning that displays in textual format either dialog of characters that are being displayed in the main media stream or sounds of events that occur in the main media stream;

(b) initializing a timing module to perform the timing of the output of the main media stream and to generate triggering events associated with the timing of the output of the main media stream;

(c) outputting the main media stream via the media stream output device;

(d) monitoring for and detecting triggering events associated with the timing of the output of the main media stream generated by the timing module; and

(e) upon detecting a triggering event;

(i) determine that a supplemental media stream is associated with the detected triggering event

and determine whether the supplemental media stream includes information indicating that the supplemental media stream is to be output synchronously or asynchronously

and upon determining that the information indicates that the supplemental media stream is to be output asynchronously, output the supplemental media stream via the media stream output device such that the output of the main media stream is automatically suspended without user intervention while the supplemental media stream is output.

46. The computer-readable medium of claim 45 further comprising, upon detecting a triggering event, determining which of a plurality of media streams are associated supplemental media streams associated with the detected triggering event, and iteratively outputting the associated supplemental media streams via the media stream output device.

47. The computer-readable medium of claim 46, wherein iteratively outputting the associated supplemental media stream via the media stream output device further comprises:

determining an order among the associated supplemental media streams; and

iteratively outputting the associated supplemental media stream via the media stream output device according to the determined order.

48. The computer-readable medium of claim 47, wherein iteratively outputting the associated supplemental media streams either synchronously or asynchronously with the main media stream further comprises suspending the output of the main media stream while an associated supplemental media stream is output asynchronously.

49. The computer-readable medium of claim 46, wherein iteratively outputting the associated supplemental media streams via the media stream output device further comprises:

determining for each associated supplemental media stream whether the associated supplemental media stream is to be output synchronously or asynchronously to the main media stream; and

iteratively outputting each associated supplemental media stream via the media stream output device either synchronously or asynchronously with the main media stream according to the results of the determinations.

50. The computer-readable medium of claim 49, wherein the supplemental media stream is an audio, visual, or compiled supplemental media stream.

51. The computer-readable medium of claim 50, wherein the compiled media stream comprises a compiled main media stream and a compiled supplemental media stream associated with the compiled main media stream, and wherein outputting the compiled media stream comprises:

asynchronously outputting the compiled main media stream to the main media stream;

monitoring for and detecting compiled triggering events while outputting the compiled main media stream; and

upon detecting a compiled triggering event;

determining that the compiled supplemental media stream is associated with the compiled triggering event

and determining whether the compiled supplemental media stream is to be output synchronously or asynchronously with the compiled main media stream; and

outputting the compiled supplemental media stream either synchronously or asynchronously with the compiled main media stream according to the results of the determination.

52. The computer-readable medium of claim 51, wherein initializing the output of the main media stream via the media stream output device comprises:

instantiating an output module for outputting the main media stream via the media stream output device according to the type of the main media stream;

binding the main media stream to the output module; and

causing the output module to output the main media stream via the media stream output device.

53. The computer-readable medium of claim 52, wherein outputting the supplemental media stream via the media stream output device further comprises:

instantiating an output module outputting the supplemental media stream via the media stream output device according to the type of the supplemental media stream;

binding the supplemental media stream to the output module; and

causing the output module to output the supplemental media stream via the media stream output device.

54. A system for outputting at least one supplemental media stream designed to assist sense-impaired people in conjunction with a main media stream, the system comprising:

(a) an output means to output a main media stream and a supplemental media stream, wherein the supplemental media stream comprises captioning that displays in textual format either dialog of characters that are being displayed in the main media stream or sounds of events that occur in the main media stream;

(b) a timing means to monitor the timing of the output of the main media stream and generate triggering events associated with the timing of the output of the main media stream; and

(c) a management means communicatively coupled to the output means and to the timing means to;

(i) obtain the main media stream;

(ii) output the main media stream via the output means;

(iii) monitor for and detect triggering events associated with the timing of the output of the main media stream generated by the timing means; and

(iv) upon detecting a triggering event;

(A) determine that the supplemental media stream is associated with the triggering event

and determine whether the supplemental media stream includes information indicating that the supplemental media stream is to be output synchronously or asynchronously

and upon determining that the information indicates that the supplemental media stream is to be output asynchronously, output the supplemental media stream via the output means such that the output of the main media stream is automatically suspended without user intervention while the supplemental media stream is output.

55. The system of claim 54, wherein the management means initializes the timing means in conjunction with the output of the main media stream.

56. The system of claim 55, wherein the management means initializes the timing means with an event table, the event table comprising a plurality of event entries, wherein each event entry identifies an associated supplemental media stream and a time during the output of the main media stream when the timing means is to generate a triggering event indicating the associated supplemental media stream is to be output.

57. The system of claim 54, wherein the output means outputs a plurality of supplemental media streams, and wherein the management means, upon detecting a triggering event, determines which of a plurality of supplemental media streams are associated supplemental media streams associated with the detected triggering event, and iteratively outputs the associated supplemental media streams via the output means.

58. The system of claim 57, wherein iteratively outputting each associated supplemental media stream via the output means comprises:

determining an order among the associated supplemental media streams; and

iteratively outputting each associated supplemental media stream via the output means according to the determined order.

59. The system of claim 58, wherein determining an order among the associated supplemental media streams comprises:

obtaining a priority value associated with each associated supplemental media stream; and

ordering the associated supplemental media streams according to their associated priority values, the resultant order being the determined order.

60. The system of claim 57, wherein iteratively outputting the associated supplemental media streams via the output means comprises:

determining for each associated supplemental media stream whether the associated supplemental media stream is to be output synchronously or asynchronously to the main media stream; and

iteratively outputting each associated supplemental media stream via the output means either synchronously or asynchronously with the main media stream according to the results of the determinations.

61. The system of claim 60, wherein iteratively outputting each associated supplemental media stream either synchronously or asynchronously with the main media stream comprises suspending the output of the main media stream while an associated supplemental media stream is output asynchronously.

62. The system of claim 61, wherein iteratively outputting each associated supplemental media stream either synchronously or asynchronously with the main media stream further comprises causing the timing means to suspend the timing of the main media stream when the main media stream is suspended.

63. The system of claim 61, wherein the timing means further monitors the timing of each associated supplemental media stream being output.

64. The system of claim 63, wherein the management means initializes the timing means in conjunction with the output of each associated supplemental media stream.

65. The system of claim 61, wherein the supplemental media stream is an audio, visual, or compiled supplemental media stream.

66. The system of claim 65, wherein the compiled media stream comprises a compiled main media stream and a compiled supplemental media stream associated with the compiled main media stream, and wherein outputting the compiled media stream comprises:

asynchronously outputting the compiled main media stream to the main media stream;

monitoring for and detecting compiled triggering events while outputting the compiled main media stream; and

upon detecting a compiled triggering event, determining that the compiled supplemental media stream is associated with the compiled triggering event

and determining whether the compiled supplemental media stream is to be output synchronously or asynchronously with the compiled main media stream; and

outputting the compiled supplemental media stream either synchronously or asynchronously with the compiled main media stream according to the results of the determination.

67. The system of claim 54, wherein the management means outputs the main media stream via the output means comprises:

instantiating a stream output means for outputting the main media stream via the output means according to the type of the main media stream;

binding the main media stream to the stream output means; and

causing the stream output means to output the main media stream via the output means.

68. The system of claim 67, wherein the management means outputs the supplemental media stream via the output means comprises:

instantiating a stream output means for outputting the supplemental media stream via the output means according to the type of the supplemental media stream;

binding the supplemental media stream to the stream output means; and

causing the stream output means to output the supplemental media stream via the output means.