Production of a video stream with synchronized annotations over a computer network

• US 6,006,241 A
• Filed: 03/14/1997
• Issued: 12/21/1999
• Est. Priority Date: 03/14/1997
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gupta, Anoop, del Val, David, Vellanki, Srinivas Prasad, Chaddha, Navin, Wang, Edward Yan-bing, Purnaveja, Audi
• Primary Examiner(s): Feild, Joseph H.
• Assistant Examiner(s): Kindred, Alford W.
• Application Number:
  US08/818,804
• Time in Patent Office: 1,012 Days
• Field of Search: 707/513, 707/517, 707/530, 707/523, 707/511, 707/512, 707/500, 707/526, 345/328, 345/329, 345/330, 345/340, 345/302
• US Class Current: 715/205
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4886 for displaying a ticker, e.g. scrolling banner for news, stock exchange, weather data
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/854 Content authoring
• CPC Class Codes: H04N 21/8586 by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 5/445 for displaying additional information H04N5/50 takes precedence
• CPC Class Codes: H04N 5/44504 Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits

Abstract

The production of synchronization scripts and associated annotated multimedia streams for servers and client computers coupled to each other by a diverse computer network which includes local area networks (LANs) and/or wide area networks (WANs) such as the internet. Annotated multimedia streams can include a compressed video stream for display in a video window, an accompanying compressed audio stream and annotations. Synchronization scripts include annotation streams for synchronizing the display of video streams with annotations, e.g., displayable events, such textual/graphical data in the form of HTML pages with Java applets to be displayed in one or more event windows. The producer includes a capture module and an author module for capturing video streams and generating annotation streams, respectively. The capture module compresses the video stream using a suitable compression format. Annotation streams include annotation frames which provide either pointer(s) to the event(s) of interest or include displayable data embedded within the annotation stream. Accordingly, each annotation frame includes either an event locator or an event data. In addition, each annotation frame includes an event time marker which corresponds to the time stamp(s) of associated video frame(s) within the video stream. Embedded displayable data include ticker tape data embedded within the annotation stream. Examples of event locators to displayable events include URL addresses pointing to HTML web pages. The video/audio streams and annotation streams are stored in stream server(s) for subsequent retrieval by client computer(s) in a coordinated manner, so that the client computer(s) is able to synchronously display the video frames and displayable event(s) in a video window and event window(s), respectively. In one implementation, annotation streams include a flipper stream for locating HTML pages and a ticker stream which include ticker (tape) data.

Claims

1. In a computer system, a method for generating a synchronization script configured to orchestrate a synchronized display of a video stream and displayable elements on a display device of a client computer, said video stream including a plurality of video frames, each said video frame associated with a time stamp, the method comprising:

generating a first annotation stream for inclusion in said synchronization script, said first annotation stream including a first plurality of annotation frames, said first plurality of annotation frames associated with a first selected subset of said video frames, each said first plurality of annotation frames including an event locator and an event time marker, each said event locator pointing to the location of one of a first plurality of displayable elements stored in a web server coupled to said client computer, said event time markers of said first plurality of annotation frames corresponding to the respective time stamps of said first selected subset of video frames.

2. The method of claim 1 wherein each said event locator of said first plurality of annotation frames includes an URL address pointing to a HTML page.

3. The method of claim 2 further comprising:

generating a second annotation stream for inclusion in said synchronization script, said second annotation stream including a second plurality of annotation frames, said second plurality of annotation frames associated with a second selected subset of said video frames, each said second plurality of annotation frames including a corresponding one of a second plurality of displayable events and an event time marker, said time markers of said second plurality of annotation frames corresponding to the respective time stamps of said second selected subset of video frames.

4. The method of claim 3 wherein said displayable events include ticker data.

5. In a computer system, a method for generating a synchronization script configured to orchestrate a synchronized display of a video stream and displayable elements on a display device of a client computer, said video stream including a plurality of video frames, each said video frame associated with a time stamp, the method comprising:

generating at least one annotation stream for inclusion in said synchronization script, said at least one annotation stream including a plurality of annotation frames, said annotation frames associated with a selected subset of said video frames, each said annotation frame including a corresponding one of said displayable events and an event time marker, said time markers of said annotation frames corresponding to the respective time stamps of said selected subset of video frames.

6. The method of claim 5 wherein said displayable events include ticker data.

7. In a client computer having a display device, a method of skipping forward or backward to a predetermined one of a plurality of video frames of a video stream being displayed on said display device, each said video frame including a time stamp, and wherein a table of contents is also displayed on said display device, the method comprising:

receiving said video stream from a stream server coupled to the client computer via a computer network;

sequentially displaying said video frames on said display device, starting from an initial video frame of said video stream;

selecting a content label from said table of contents, said content label providing an index into an intermediate video frame of said video stream, said index based on the time stamp of said intermediate video frame;

communicating said index to said stream server;

receiving said video stream from said stream server, starting from said intermediate video frame; and

sequentially displaying said video frames on said display device, starting with said intermediate video frame.

8. In a stream server, a method of causing a client computer to skip forward or backward to a predetermined one of a plurality of video frames of a video stream being displayed on a display device of said client computer, said stream server coupled to the client computer via a computer network, each said video frame including a time stamp, and wherein a table of content is also displayed on said display device, the method comprising:

streaming said video stream to said client computer for sequential display on said display device, starting from an initial video frame of said video stream;

receiving an index from said client computer, said index associated with a selectable content label from a table of contents displayed on said display device, said index based on the time stamp of an intermediate video frame of said video stream; and

streaming said video stream to said client computer for sequential display on said display device, starting from said intermediate video frame.

9. A producer useful for generating a synchronization script configured to orchestrate a synchronized display of a video stream and displayable elements on a display device of a client computer, the producer comprising:

a capture module configured to capture said video stream which includes a plurality of video frames, each said video frame associated with a time stamp; and

an author module configured to generate a first annotation stream for inclusion in said synchronization script, said first annotation stream including a first plurality of annotation frames, said first plurality of annotation frames associated with a first selected subset of said video frames, each said first plurality of annotation frames including an event locator and an event time marker, each said event locator pointing to the location of one of a first plurality of displayable elements stored in a web server coupled to said client computer, said event time markers of said first plurality of annotation frames corresponding to the respective time stamps of said first selected subset of video frames.

10. The producer of claim 9 wherein each said event locator of said first plurality of annotation frames includes an URL address pointing to a HTML page.

11. The producer of claim 9 wherein said author module is further configured to generate a second annotation stream for inclusion in said synchronization script, said second annotation stream including a second plurality of annotation frames, said second plurality of annotation frames associated with a second selected subset of said video frames, each said second plurality of annotation frames including a corresponding one of a second plurality of displayable events and an event time marker, said time markers of said second plurality of annotation frames corresponding to the respective time stamps of said second selected subset of video frames.

12. The producer of claim 11 wherein said displayable events include ticker data.

13. A producer useful for generating a synchronization script configured to orchestrate a synchronized display of a video stream and displayable elements on a display device of a client computer, the producer comprising:

a capture module configured to capture said video stream which includes a plurality of video frames, each said video frame associated with a time stamp; and

an author module configured to generate at least one annotation stream for inclusion in said synchronization script, said at least one annotation stream including a plurality of annotation frames, said annotation frames associated with a selected subset of said video frames, each said annotation frame including a corresponding one of said displayable events and an event time marker, said time markers of said annotation frames corresponding to the respective time stamps of said selected subset of video frames.

14. The producer of claim 13 wherein said displayable events include ticker data.

15. A client computer useful in association with a stream server coupled to the client computer via a computer network, the client computer comprising:

a playout buffer configured to receive a video stream from said stream server, said video stream including a plurality of video frames, each said video frame including a time stamp;

a display device configured to sequentially display said video frames on said display device, starting from an initial video frame of said video stream, said display device also configured to display a selectable content label from said table of contents, said selectable content label providing an index into an intermediate video frame of said video stream, said index based on the time stamp of said intermediate video frame; and

a transmitter configured to communicate said index to said stream server, thereby causing said stream server to begin transmitting said video stream to said client computer, starting from said intermediate video frame, and causing said display device to sequentially display said video frames, starting with said intermediate video frame.

16. A stream server useful in association with a client computer coupled to said stream server via a computer network, the stream server comprising:

a streamer configured to store and stream a video stream to said client computer, said video stream including a plurality of video frames, said video stream enabling said client computer to sequentially display said video frames on a display device of said client computer, starting from an initial video frame of said video stream; and

a receiver configured to receive an index from said client computer, said index associated with a selectable content label from a table of contents displayed on said display device, said index based on the time stamp of an intermediate video frame of said video stream, and wherein said received index causes said streamer to begin streaming said video stream to said client computer, and enabling said client computer to sequentially display said video frames on said display device starting from said intermediate video frame.

17. A computer-readable medium useful in association with a computer system, the computer-readable medium having computer-executable instructions which cause the computer system to perform a method comprising generating a first annotation stream which includes a first plurality of annotation frames, said first plurality of annotation frames associated with a first selected subset of a plurality of video frames of a video stream, each said video frame associated with a time stamp, each said first plurality of annotation frames including an event locator and an event time marker, each said event locator pointing to the location of one of a first plurality of displayable elements stored in a web server coupled to said client computer, said event time markers of said first plurality of annotation frames corresponding to the respective time stamps of said first selected subset of video frames.

18. The computer-readable medium of claim 17 wherein each said event locator of said first plurality of annotation frames includes an URL address pointing to a HTML page.

19. The computer-readable medium of claim 18 having further computer-readable code instructions comprising:

generating a second annotation stream which includes a second plurality of annotation frames, said second plurality of annotation frames associated with a second selected subset of said video frames, each said second plurality of annotation frames including a corresponding one of a second plurality of displayable events and an event time marker, said time markers of said second plurality of annotation frames corresponding to the respective time stamps of said second selected subset of video frames.

20. The computer-readable medium of claim 19 wherein said displayable events include ticker data.

21. A computer-readable medium useful in association with a computer system, the computer-readable medium having computer-executable instructions which cause the computer system to perform a method comprising generating at least one annotation stream which includes a plurality of annotation frames, said annotation frames associated with a selected subset of a plurality of video frames of a video stream, each said video frame associated with a time stamp, each said annotation frame including a corresponding one of said displayable events and an event time marker, said time markers of said annotation frames corresponding to the respective time stamps of said selected subset of video frames.

22. The computer-readable medium of claim 21 wherein said displayable events include ticker data.