Dynamic client-server video tiling streaming

• US 8,630,512 B2
• Filed: 01/25/2008
• Issued: 01/14/2014
• Est. Priority Date: 01/25/2007
• Status: Active Grant

• Current Assignee: Otello Corp. ASA
• Original Assignee: Skyfire Labs Inc. (Otello Corp. ASA)
• Inventors: Swenson, Erik R., Bhandari, Nitin, Kumar, Vijay
• Primary Examiner(s): Krasnic, Bernard
• Application Number:
  US12/020,361
• Publication Number:
  US 20080181498A1
• Time in Patent Office: 2,181 Days
• Field of Search: 382/305, 382/232, 382/282, 382/293, 382/299
• US Class Current: 382/305
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2340/06 Colour space transformation
• CPC Class Codes: G09G 2340/145 related to small screens
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2370/042 for monitor identification
• CPC Class Codes: G09G 2370/10 Use of a protocol of communication by packets in interfaces along the display data pipeline
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/15 by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 21/234336 by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4782 Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 21/8451 using Advanced Video Coding [AVC]

Abstract

A system and a method for communicating video data associated with content across a network are disclosed. A server partitions the content into a plurality of tiles where each tile includes a subset of the content and index data describing a relationship between a tile and one or more adjacent tiles. Each tile is then encoded as a video frame, for example each tile is encoded according to the H.264 standard. The plurality of tiles are then transmitted to a client which combines the plurality of tiles using the index data to locally recreate the content. The client associates a predetermined viewport tile with a client viewport representing the area of the content that is visible in a client display device to allow viewing of the content while tiles not associated with the client viewport are cached to expedite subsequent retrieval.

Claims

1. A method for communicating content across a network using video frames, the method comprising:

partitioning the content into a plurality of tiles, each tile including a subset of the content and index data describing a relationship between a tile and one or more adjacent tiles;

encoding the content associated with each tile in the plurality of tiles as a reference frame;

transmitting the plurality of encoded tiles to a client to allow the client to combine the plurality of encoded tiles using the index data;

responsive to a modification to the content associated with a tile, generating a difference frame, the difference frame describing the modification to the content associated with the tile in relation to the reference frame;

comparing a storage size of the difference frame to a threshold specifying a maximum storage size of the difference frame;

responsive to the storage size of the difference frame being less than the threshold, transmitting the index data of the tile whose content has been modified and the difference frame to the client, the client configured to combine the difference frame with the reference frame associated with the tile identified by the index data; and

responsive to the storage size of the difference frame being greater than the threshold,

generating a replacement reference frame, the replacement reference frame including the modified content associated with the tile, and

transmitting the index data of the tile whose content has been modified and the replacement reference frame to the client to allow the client to combine the replacement reference frame with one or more of the plurality of encoded tiles using the index data.

2. The method of claim 1, wherein the index data further comprises a tile identifier, a tile height and width, or clipping data, the clipping data describing editing of the content to not exceed the tile height and width.

3. The method of claim 1, wherein partitioning the content into a plurality of tiles comprises:

segmenting the content into a grid, the grid comprising a plurality of tiles having a fixed height and width and each tile adjacent to one or more surrounding tiles.

4. The method of claim 1, further comprising:

caching the reference frame associated with each tile in the plurality of tiles.

5. The method of claim 1, further comprising:

caching the difference frame and the index data associating the difference frame with the reference frame including the modification to content.

6. A non-transitory computer-readable storage medium configured to store instructions for communicating content across a network using video frames, the instructions when executed by a processor causing the processor to:

partition the content into a plurality of tiles, each tile including a subset of the content and index data describing a relationship between a tile and one or more adjacent tiles;

encode the content associated with each tile in the plurality of tiles as a reference frame;

transmit the plurality of encoded tiles to a client to allow the client to combine the plurality of encoded tiles using the index data;

responsive to a modification to the content associated with a tile, generate a difference frame, the difference frame describing the modification to the content associated with the tile in relation to the reference frame;

compare a storage size of the difference frame to a threshold specifying a maximum storage size of the difference frame;

responsive to the storage size of the difference frame being less than the threshold, transmit the index data of the tile whose content has been modified and the difference frame to the client, the client configured to combine the difference frame with the reference frame associated with the tile identified by the index data; and

responsive to the storage size of the difference frame being greater than the threshold,

generate a replacement reference frame, the replacement reference frame including the modified content associated with the tile, and

transmit the index data of the tile whose content has been modified and the replacement reference frame to the client to allow the client to combine the replacement reference frame with one or more of the plurality of encoded tiles using the index data.

7. The non-transitory computer-readable storage medium of claim 6, wherein the index data further comprises a tile identifier, a tile height and width, or clipping data, the clipping data describing editing of the content to not exceed the tile height and width.

8. The non-transitory computer-readable storage medium of claim 6, wherein the instructions that when executed cause the processor to partition the content into a plurality of tiles further comprises instructions that when executed cause the processor to segment the content into a grid, the grid comprising a plurality of tiles having a fixed height and width and each tile adjacent to one or more surrounding tiles.

9. The non-transitory computer-readable storage medium of claim 6, further comprising instructions that when executed cause the processor to cache the reference frame associated with each tile in the plurality of tiles.

10. The non-transitory computer-readable storage medium of claim 6, further comprising instructions that when executed cause the processor to cache the difference frame and the index data associating the difference frame with the reference frame including the modification to content.

11. A system for communicating content across a network using video frames, the system comprising:

a processor configured to execute computer program instructions; and

a non-transitory computer readable storage medium storing the computer program instructions executable by the processor, the instructions for;

partitioning the content into a plurality of tiles, each tile including a subset of the content and index data describing a relationship between a tile and one or more adjacent tiles;

encoding the content associated with each tile in the plurality of tiles as a reference frame;

transmitting the plurality of encoded tiles to a client to allow the client to combine the plurality of encoded tiles using the index data;

responsive to a modification to the content associated with a tile, generating a difference frame, the difference frame describing the modification to the content associated with the tile in relation to the reference frame;

comparing a storage size of the difference frame to a threshold specifying a maximum storage size of the difference frame;

responsive to the storage size of the difference frame being less than the threshold, transmitting the index data of the tile whose content has been modified and the difference frame to the client, the client configured to combine the difference frame with the reference frame associated with the tile identified by the index data; and

responsive to the storage size of the difference frame being greater than the threshold;

generating a replacement reference frame, the replacement reference frame including the modified content associated with the tile, and

transmitting the index data of the tile whose content has been modified and the replacement reference frame to the client to allow the client to combine the replacement reference frame with one or more of the plurality of encoded tiles using the index data.

12. The system of claim 11, wherein the index data further comprises a tile identifier, a tile height and width, or clipping data, the clipping data describing editing of the content to not exceed the tile height and width.

13. The system of claim 11, wherein the instructions for partitioning the content into a plurality of tiles further comprises instructions for segmenting the content into a grid, the grid comprising a plurality of tiles having a fixed height and width and each tile adjacent to one or more surrounding tiles.

14. The system of claim 11, further comprising instructions for caching the reference frame associated with each tile in the plurality of tiles.

15. The system of claim 11, further comprising instructions for caching the difference frame and the index data associating the difference frame with the reference frame including the modification to content.