Integrating continuous and sparse streaming data

• US 10,257,587 B2
• Filed: 08/22/2016
• Issued: 04/09/2019
• Est. Priority Date: 10/06/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bocharov, John A., Zhang, Geqiang (Sam), Duggaraju, Krishna Prakash, Liu, Lin, Roy, Anirban, Freelander, Jack E., Sood, Vishal
• Primary Examiner(s): Lewis, Jonathan V
• Application Number:
  US15/243,134
• Publication Number:
  US 20160360297A1
• Time in Patent Office: 960 Days
• Field of Search: None
• CPC Class Codes: H04N 21/2183 Cache memory caches in web servers or browsers G06F16/9574; intermediate storage and caching in data networks H04L67/568
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/858 Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

A sparse streaming system provides a first-class means for sparse metadata to be added to streaming media presentations and to be delivered using an integrated data channel that is cacheable using readily available HTTP-based Internet caching infrastructure for increased scalability. The sparse streaming system stores a reference to a sparse track within a continuous track. If a continuous fragment arrives at the client that refers to a sparse fragment that the client has not yet retrieved, then the client requests the sparse fragment. In addition, each sparse fragment may include a backwards reference to the sparse fragment created immediately prior. The references in the continuous fragments make the client aware of new sparse track fragments, and the backwards references in the sparse track fragments ensure that the client has not missed any intervening sparse track fragments.

Claims

1. A computer-implemented method for controlling a client computer system to request sparse track fragments associated with a continuous data track, comprising:

requesting, from a server, a continuous track fragment associated with the continuous data track of a media presentation over a network;

receiving the continuous track fragment over the network;

identifying a reference to a sparse track fragment associated with the continuous track fragment;

determining that the sparse track fragment has not previously been received;

in response to the determining that the sparse track fragment has not previously been received, requesting the sparse track fragment over the network;

receiving the sparse track fragment over the network;

identifying a reference to a previous sparse track fragment in a header associated and transmitted with the sparse track fragment;

determining that the previous sparse track fragment has not previously been received;

in response to the determining that the previous sparse track fragment has not previously been received, retrieving the previous sparse track fragment over the network; and

displaying the continuous track fragment, the sparse track fragment, and the previous sparse track fragment, wherein the sparse track fragment and the previous sparse track fragment are superimposed on the continuous data track.

2. The method of claim 1, wherein identifying the reference to the sparse track fragment associated with the continuous track fragment comprises:

identifying the reference to the sparse track fragment in a header associated with continuous track fragment.

3. The method of claim 1, wherein determining that the sparse track fragment has not been received comprises:

comparing a track identifier and time stamp associated with the sparse track fragment with a respective track identifier and time stamp associated with at least one previously-received sparse track fragment.

4. The method of claim 1, wherein receiving the continuous track fragment comprises:

receiving error recovery information that allows discovering missed sparse track fragments by following a stored reference between a current sparse track fragment and a previous sparse track fragment without retrieving intervening continuous track fragments.

5. The method of claim 1, wherein receiving the continuous track fragment comprises:

receiving the continuous track fragment from a server cache.

6. The method of claim 1, wherein receiving the continuous track fragment comprises receiving a video fragment;

and

wherein receiving the sparse track fragment comprises receiving a text fragment.

7. The method of claim 1, wherein the sparse track fragment includes spatial information that specifies an insertion point in the continuous track.

8. A client computer system for requesting sparse track fragments associated with a continuous data track of a media presentation, the system comprising:

a processor;

a memory comprising software, the software including instructions to cause the processor to perform the actions of;

requesting, from a server, a continuous track fragment associated with the continuous data track over a network;

receiving the continuous track fragment over the network;

identifying a reference to a sparse track fragment associated with the continuous track fragment;

determining that the sparse track fragment has not previously been received;

in response to the determining that the sparse track fragment has not previously been received, requesting the sparse track fragment over the network;

receiving the sparse track fragment over the network;

identifying a reference to a previous sparse track fragment in a header associated and transmitted with the sparse track fragment;

determining that the previous sparse track fragment has not previously been received;

in response to the determining that the previous sparse track fragment has not previously been received, retrieving the previous sparse track fragment over the network; and

displaying the continuous track fragment, the sparse track fragment, and the previous sparse track fragment, wherein the sparse track fragment and the previous sparse track fragment are superimposed on the continuous data track.

9. The system of claim 8, wherein the software further includes instructions to cause the processor to perform the action of:

identifying the reference to the sparse track fragment associated with the continuous track fragment by identifying the reference to the sparse track fragment in a header associated with continuous track fragment.

10. The system of claim 8, wherein the software further includes instructions to cause the processor to perform the action of:

determining that the sparse track fragment has not been received by comparing a track identifier and time stamp associated with the sparse track fragment with a respective track identifier and time stamp associated with at least one previously-received sparse track fragment.

11. The system of claim 8, wherein receiving the continuous track fragment comprises:

receiving error recovery information that allows discovering missed sparse track fragments by following a stored reference between a current sparse track fragment and a previous sparse track fragment without retrieving intervening continuous track fragments.

12. The system of claim 8, wherein the software further includes instructions to cause the processor to perform the action of:

receiving the continuous track fragment from a server cache.

13. The system of claim 8, wherein the continuous track fragment comprises a video fragment and the sparse track fragment comprises a text fragment.

14. The system of claim 8, wherein the software further includes instructions to cause the processor to perform the action of:

regularly requesting continuous track fragments according to one or more of a specified time interval or a quality of service (QoS) data.

15. The system of claim 8, wherein the software further includes instructions to cause the processor to perform the action of:

requesting the sparse track fragment by issuing a Hypertext Transfer Protocol (HTTP) request that specifies a Uniform Resource Locator (URL) associated with the sparse track fragment.

16. The system of claim 8, wherein the sparse track fragment includes spatial information that specifies an insertion point in the continuous track.

17. A computer-readable storage device comprising instructions for controlling a client computer system to request sparse track fragments associated with a continuous data track, wherein the instructions, when executed, cause a processor to perform actions comprising:

requesting, from a server, a continuous track fragment associated with the continuous data track of a media presentation over a network;

receiving the continuous track fragment over the network;

identifying a reference to a sparse track fragment associated with the continuous track fragment;

determining that the sparse track fragment has not previously been received;

in response to the determining that the sparse track fragment has not previously been received, requesting the sparse track fragment over the network;

receiving the sparse track fragment over the network;

identifying a reference to a previous sparse track fragment in a header associated and transmitted with the sparse track fragment;

determining that the previous sparse track fragment has not previously been received;

in response to the determining that the previous sparse track fragment has not previously been received, retrieving the previous sparse track fragment over the network; and

displaying the continuous track fragment, the sparse track fragment, and the previous sparse track fragment, wherein the sparse track fragment and the previous sparse track fragment are superimposed on the continuous data track.

18. The computer-readable storage device of claim 17 wherein the sparse track fragment includes spatial information that specifies an insertion point in the continuous track.

19. The computer-readable storage device of claim 17, wherein determining that the sparse track fragment has not been received comprises:

comparing a track identifier and time stamp associated with the sparse track fragment with a respective track identifier and time stamp associated with at least one previously-received sparse track fragment.

20. The computer-readable storage device of claim 17, wherein the actions further comprise:

receiving error recovery information that allows discovering missed sparse track fragments by following a stored reference between a current sparse track fragment and a previous sparse track fragment without retrieving intervening continuous track fragments.