Streaming and displaying a video stream with synchronized annotations over a computer network

• US 6,173,317 B1
• Filed: 03/14/1997
• Issued: 01/09/2001
• Est. Priority Date: 03/14/1997
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Edward, Dean, Dawson, Vellanki, Prasad, Gupta, Anoop, del Val, David, Cannon, Anthony, Chaddha, Navin
• Primary Examiner(s): Kostak, Victor R.
• Application Number:
  US08/819,585
• Time in Patent Office: 1,397 Days
• Field of Search: 348/7, 348/10, 348/12, 348/13, 345/327, 345/328, 455/4.2, 455/6.3, 455/5.1, 709/217-219
• US Class Current: 709/219
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/4886 for displaying a ticker, e.g. scrolling banner for news, stock exchange, weather data
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8455 involving pointers to the content, e.g. pointers to the I-frames of the video stream
• CPC Class Codes: H04N 21/858 Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
• CPC Class Codes: H04N 21/8586 by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02
• CPC Class Codes: H04N 7/165 : Centralised control of user terminal subsequent to an upstream request signal H04N7/17345; Registering at central by two-way working H04N7/17309
• CPC Class Codes: H04N 7/17318 : Direct or substantially direct transmission and handling of requests

Abstract

Client computer(s) retrieve and display synchronized annotated multimedia streams from servers dispersed over a diverse computer network which includes local area networks (LANs) and/or wide area networks (WANs) such as the internet. Multimedia streams provided to the client computer(s) can include a compressed video stream for display in a video window and an accompanying compressed audio stream. Annotations, i.e., displayable events, include textual/graphical data in the form of HTML pages with Java applets to be displayed in one or more event windows. The video/audio and annotation streams are produced and then stored in stream server(s). Annotation streams include annotation frames which provide either pointer(s) to the event(s) of interest or include displayable data embedded within the annotation stream. Accordingly, each annotation frame includes either an event locator or an event data. In addition, each annotation frame includes an event time marker which corresponds to the time stamp(s) of associated video frame(s) within the video stream. Examples of embedded displayable data include ticker tape data embedded within the annotation stream. Examples of event locators to displayable events include URL addresses pointing to HTML web pages. Video/audio streams and annotation streams are provided by the stream server(s) to the client computer(s) in a coordinated manner, so that the client computer(s) is able to synchronously display the video frames and displayable event(s) in a video window and event window(s), respectively.

Claims

1. In a client computer having a processor, memory and a display device, said client computer coupled to a stream server and at least one web server via a network, a method for synchronizing the display of video frames of a video stream with the display of displayable events, said method comprising the steps of:

receiving a plurality of video frames of said video stream from said stream server for display in a video window of said display device, each said video frame associated with a time stamp;

displaying said video frames in said video window;

receiving a first plurality of annotation frames of a first annotation stream from said stream server;

interpreting said first plurality of annotation frames which is associated with a first selected subset of said video frames, each of said first plurality of annotation frames including an event locator and an event time marker, and wherein said event time markers of said first plurality of annotation frames corresponds to the respective time stamps of said first selected subset of video frames, and each said event locator points to one of a first plurality of displayable events stored in said at least one web server; and

retrieving and synchronously displaying in a first event window of said display device the displayable event pointed to by the event locator of each of said first plurality of annotation frames whenever the event time marker of each of said first plurality of annotation frames matches the time stamp associated with the video frame currently being displayed in said video window.

2. The method of claim 1 wherein each said event locator of said first plurality of annotation frames is an URL address pointing to a HTML page.

3. The method of claim 1 further comprising the steps of:

receiving a second plurality of annotation frames of a second annotation stream from said stream server;

interpreting said second plurality of annotation frames, said second plurality of annotation frames associated with a second selected subset of said video frames, each of said second plurality of annotation frames including a corresponding one of a second plurality of displayable events and an event time marker, and wherein said event time markers of said second plurality of annotation frames corresponds to the respective time stamps of said second selected subset of video frames; and

synchronously displaying in a second event window of said display device said one corresponding displayable event whenever the event time marker of each of said second plurality of annotation frames matches the time stamp associated with the video frame currently being displayed in said video window.

4. The method of claim 1 wherein said video server and said web server are part of a server computer system.

5. The method of claim 3 wherein said second plurality of displayable events include ticker data.

6. In a stream server having a processor and memory, said stream server coupled to a client computer and at least one web server via a network, said client computer having a processor, memory and a display device, a method for streaming video and annotation streams to synchronize the display of video frames of a video stream with the display of displayable events, said method comprising the steps of:

streaming a plurality of video frames of said video stream from said stream server to said client computer for display in a video window of said display device, each said video frame associated with a time stamp; and

synchronously streaming a first plurality of annotation frames of a first annotation stream from said stream server to said client computer, said first plurality of annotation frames associated with a first selected subset of said video frames, each of said first plurality of annotation frames including an event locator and an event time marker, and wherein each said event locator points to one of a first plurality of displayable events stored in said at least one web server, said event time markers of said first plurality of annotation frames corresponds to the respective time stamps of said first selected subset of video frames, and said video stream and said first annotation stream are synchronized using said respective time stamps of said first selected subset of said video frames and said event time markers of said first plurality of annotation frames.

7. The method of claim 6 wherein each said event locator of said plurality of annotation frames is an URL address pointing to a HTML page.

8. The method of claim 6 wherein comprising the step of:

synchronously streaming a second plurality of annotation frames of a second annotation stream from said stream server to said client computer, said second plurality of annotation frames associated with a second selected subset of said video frames, each of said second plurality of annotation frames including a corresponding one of a second plurality of displayable events and an event time marker, and wherein said event time markers of said second plurality of annotation frames corresponds to the respective time stamps of said second selected subset of video frames, and said video stream and said annotation stream are synchronized using said respective time stamps of said second selected subset of said video frames and said event time markers of said second plurality of annotation frames.

9. The method of claim 8 wherein said second plurality of displayable events include ticker data.

10. In a client computer having a processor, memory and a display device, said client computer coupled to a stream server via a network, a method for synchronizing the display of video frames of a video stream and the display of a plurality of displayable events, wherein said video stream and said plurality of displayable events are to be displayed concurrently in a video window and an event window, respectively, of said display device, said method comprising the steps of:

receiving a plurality of video frames of said video stream from said stream server for display in said video window, each said video frame associated with a time stamp;

displaying said video frames in said video window;

receiving a plurality of annotation frames of an annotation stream from said stream server;

interpreting said plurality of annotation frames, said plurality of annotation frames associated with a selected subset of said video frames, each of said plurality of annotation frames including a corresponding one of said displayable events and an event time marker, and wherein said event time markers of said plurality of annotation frames corresponds to the respective time stamps of said selected subset of video frames; and

synchronously displaying in said event window said one corresponding displayable event whenever the event time marker of each of said plurality of annotation frames matches the time stamp associated with the video frame currently being displayed in said video window.

11. The method of claim 10 wherein said displayable events include ticker data.

12. In a stream server having a processor and memory, said stream server coupled to a client computer via a network, said client computer having a processor, memory and a display device, a method for streaming video and annotation streams to synchronize the display of video frames of said video stream with the display of a plurality of displayable events, wherein said video stream and said plurality of displayable events are to be displayed concurrently in a video window and a first event window, respectively, of said display device, said method comprising the steps of:

streaming a plurality of video frames of said video stream from said stream server to said client computer for display in said video window, each said video frame associated with a time stamp; and

synchronously streaming a plurality of annotation frames of an annotation stream from said stream server to said client computer, said plurality of annotation frames associated with a selected subset of said video frames, each of said plurality of annotation frames including a corresponding one of said displayable events and an event time marker, and wherein said event time markers of said plurality of annotation frames corresponds to the respective time stamps of said selected subset of video frames, and said video stream and said annotation stream are synchronized using said respective time stamps and said event time markers.

13. The method of claim 12 wherein said displayable events include ticker data.

14. A client computer useful in association with a stream server and at least one web server, said client computer comprising:

a playout buffer configured to receive a plurality of video frames of a video stream from said stream server, each said video frame associated with a time stamp;

an annotation interpreter configured to receive and interpret a first plurality of annotation frames of a first annotation stream from said stream server, said first plurality of annotation frames associated with a first selected subset of said video frames, each of said first plurality of annotation frames including an event locator and an event time marker, and wherein said event time markers of said first plurality of annotation frames corresponds to the respective time stamps of said first selected subset of video frames, and each said event locator points to one of a first plurality of displayable events stored in said at least one web server; and

a display device configured to display said video frames in a video window of said display device and further configured to synchronously display in a first event window of said display device the displayable event pointed to by the event locator of each of said first plurality of annotation frames whenever the event time marker of each of said first plurality of annotation frames matches the time stamp associated with the video frame currently being displayed in said video window.

15. The client computer of claim 14 wherein each said event locator of said first plurality of annotation frames is an URL address pointing to a HTML page.

16. The client computer of claim 15 wherein:

said annotation interpreter is further configured to interpret a second plurality of annotation frames of a second annotation stream from said stream server, said second plurality of annotation frames associated with a second selected subset of said video frames, each of said second plurality of annotation frames including a corresponding one of a second plurality of displayable events and an event time marker, and wherein said event time markers of said second plurality of annotation frames corresponds to the respective time stamps of said second selected subset of video frames; and

said display device is further configured to synchronously display in a second event window of said display device said one corresponding displayable event whenever the event time marker of each of said second plurality of annotation frames matches the time stamp associated with the video frame currently being displayed in said video window.

17. The client computer of claim 14 wherein said video server and said web server are part of a server computer system.

18. The client computer of claim 16 wherein said second plurality of displayable events include ticker data.

19. A stream server useful in association with a client computer and at least one web server, said client computer having a display device, the stream server comprising:

memory configured to store a plurality of video frames of said video stream to be streamed from said stream server to said client computer for display in a video window of said display device, each said video frame associated with a time stamp; and

wherein said memory is further configured to store a first plurality of annotation frames of a first annotation stream to be synchronously streamed from said stream server to said client computer, said first plurality of annotation frames associated with a first selected subset of said video frames, each of said first plurality of annotation frames including an event locator and an event time marker, and wherein each said event locator points to one of a first plurality of displayable events stored in said at least one web server, said event time markers of said first plurality of annotation frames corresponds to the respective time stamps of said first selected subset of video frames, and said video stream and said first annotation stream are synchronized using said respective time stamps of said first selected subset of said video frames and said event time markers of said first plurality of annotation frames.

20. The stream server of claim 19 wherein each said event locator of said plurality of annotation frames is an URL address pointing to a HTML page.

21. The stream server of claim 19 wherein said memory is further configured to store a second plurality of annotation frames of a second annotation stream to be synchronously streamed from said stream server to said client computer, said second plurality of annotation frames associated with a second selected subset of said video frames, each of said second plurality of annotation frames including a corresponding one of a second plurality of displayable events and an event time marker, and wherein said event time markers of said second plurality of annotation frames corresponds to the respective time stamps of said second selected subset of video frames, and said video stream and said annotation stream are synchronized using said respective time stamps of said second selected subset of said video frames and said event time markers of said second plurality of annotation frames.

22. The stream server of claim 21 wherein said second plurality of displayable events include ticker data.

23. A client computer useful in association with a stream server, said client computer comprising:

a playout buffer configured to receive a plurality of video frames of said video stream from said stream server, each said video frame associated with a time stamp;

an annotation interpreter configured to receive and interpret a plurality of annotation frames of an annotation stream from said stream server, said plurality of annotation frames associated with a selected subset of said video frames, each of said plurality of annotation frames including a corresponding one of said displayable events and an event time marker, and wherein said event time markers of said plurality of annotation frames corresponds to the respective time stamps of said selected subset of video frames; and

a display device configured to display said video frames in a video window of said display device, and further configured to synchronously display in an event window of said display device said one corresponding displayable event whenever the event time marker of each of said plurality of annotation frames matches the time stamp associated with the video frame currently being displayed in said video window.

24. The client computer of claim 23 wherein said displayable events include ticker data.

25. A stream server useful in association with a client computer, said client computer having a display device, the stream server comprising:

memory configured to store a plurality of video frames of said video stream to be streamed from said stream server to said client computer for display in a video window of said display device, each said video frame associated with a time stamp; and

wherein said memory is further configured to store a plurality of annotation frames of an annotation stream to be synchronously streamed from said stream server to said client computer, said plurality of annotation frames associated with a selected subset of said video frames, each of said plurality of annotation frames including a corresponding one of a plurality of displayable events and an event time marker, and wherein said event time markers of said plurality of annotation frames corresponds to the respective time stamps of said selected subset of video frames, and said video stream and said annotation stream are synchronized using said respective time stamps of said selected subset of said video frames and said event time markers of said plurality of annotation frames.

26. The stream server of claim 25 wherein said displayable events include ticker data.

27. A computer-readable medium useful in association with a client computer having a processor, memory and a display device, said client computer coupled to a stream server and at least one web server via a network, the computer-readable medium comprising computer-readable code instructions configured to cause said client computer to execute the steps of:

receiving a plurality of video frames of a video stream from said stream server for display in a video window of said display device, each said video frame associated with a time stamp;

displaying said video frames in said video window;

receiving a first plurality of annotation frames of a first annotation stream from said stream server;

interpreting said first plurality of annotation frames which is associated with a first selected subset of said video frames, each of said first plurality of annotation frames including an event locator and an event time marker, and wherein said event time markers of said first plurality of annotation frames corresponds to the respective time stamps of said first selected subset of video frames, and each said event locator points to one of a first plurality of displayable events stored in said at least one web server; and

retrieving and synchronously displaying in a first event window of said display device the displayable event pointed to by the event locator of each of said first plurality of annotation frames whenever the event time marker of each of said first plurality of annotation frames matches the time stamp associated with the video frame currently being displayed in said video window.

28. The computer-readable medium of claim 27 wherein each said event locator of said first plurality of annotation frames is an URL address pointing to a HTML page.

29. The computer-readable medium of claim 27 further comprising computer-readable code instructions configured to cause said client computer to execute the steps of:

receiving a second plurality of annotation frames of a second annotation stream from said stream server;

interpreting said second plurality of annotation frames, said second plurality of annotation frames associated with a second selected subset of said video frames, each of said second plurality of annotation frames including a corresponding one of a second plurality of displayable events and an event time marker, and wherein said event time markers of said second plurality of annotation frames corresponds to the respective time stamps of said second selected subset of video frames; and

synchronously displaying in a second event window of said display device said one corresponding displayable event whenever the event time marker of each of said second plurality of annotation frames matches the time stamp associated with the video frame currently being displayed in said video window.

30. The computer-readable medium of claim 27 wherein said video server and said web server are part of a server computer system.

31. The computer-readable medium of claim 29 wherein said second plurality of displayable events include ticker data.

32. A computer-readable medium useful in association with a stream server having a processor and memory, said stream server coupled to a client computer and at least one web server via a network, said client computer having a processor, memory and a display device, the computer-readable medium comprising computer-readable code instructions configured to cause said stream server to execute the steps of:

streaming a plurality of video frames of a video stream from said stream server to said client computer for display in a video window of said display device, each said video frame associated with a time stamp; and

synchronously streaming a first plurality of annotation frames of a first annotation stream from said stream server to said client computer, said first plurality of annotation frames associated with a first selected subset of said video frames, each of said first plurality of annotation frames including an event locator and an event time marker, and wherein each said event locator points to one of a first plurality of displayable events stored in said at least one web server, said event time markers of said first plurality of annotation frames corresponds to the respective time stamps of said first selected subset of video frames, and said video stream and said first annotation stream are synchronized using said respective time stamps of said first selected subset of said video frames and said event time markers of said first plurality of annotation frames.

33. The computer-readable medium of claim 32 wherein each said event locator of said plurality of annotation frames is an URL address pointing to a HTML page.

34. The computer-readable medium of claim 32 further comprising computer-readable code instructions configured to cause said stream server to execute the step of:

synchronously streaming a second plurality of annotation frames of a second annotation stream from said stream server to said client computer, said second plurality of annotation frames associated with a second selected subset of said video frames, each of said second plurality of annotation frames including a corresponding one of a second plurality of displayable events and an event time marker, and wherein said event time markers of said second plurality of annotation frames corresponds to the respective time stamps of said second selected subset of video frames, and said video stream and said annotation stream are synchronized using said respective time stamps of said second selected subset of said video frames and said event time markers of said second plurality of annotation frames.

35. The computer-readable medium of claim 34 wherein said second plurality of displayable events include ticker data.

36. A computer-readable medium useful in association with a client computer having a processor, memory and a display device, said client computer coupled to a stream server via a network, the computer-readable medium comprising computer-readable code instructions configured to cause said client computer to execute the steps of:

receiving a plurality of video frames of a video stream from said stream server for display in a video window of said display device, each said video frame associated with a time stamp;

displaying said video frames in said video window;

receiving a plurality of annotation frames of an annotation stream from said stream server;

interpreting said plurality of annotation frames, said plurality of annotation frames associated with a selected subset of said video frames, each of said plurality of annotation frames including a corresponding one of a plurality of displayable events and an event time marker, and wherein said event time markers of said plurality of annotation frames corresponds to the respective time stamps of said selected subset of video frames; and

synchronously displaying in an event window of said display device said one corresponding displayable event whenever the event time marker of each of said plurality of annotation frames matches the time stamp associated with the video frame currently being displayed in said video window.

37. The client computer of claim 36 wherein said displayable events include ticker data.

38. A computer-readable medium useful in association with a stream server having a processor and memory, said stream server coupled to a client computer via a network, said client computer having a processor, memory and a display device, the computer-readable medium comprising computer-readable code instructions configured to cause said stream server to execute the steps of:

streaming a plurality of video frames of a video stream from said stream server to said client computer for display in a video window of said display device, each said video frame associated with a time stamp; and

synchronously streaming a plurality of annotation frames of an annotation stream from said stream server to said client computer, said plurality of annotation frames associated with a selected subset of said video frames, each of said plurality of annotation frames including a corresponding one of said displayable events and an event time marker, and wherein said event time markers of said plurality of annotation frames corresponds to the respective time stamps of said selected subset of video frames, and said video stream and said annotation stream are synchronized using said respective time stamps and said event time markers.

39. The stream server of claim 38 wherein said displayable events include ticker data.