Method and system for subject video streaming

• US 20030172131A1
• Filed: 05/12/2003
• Published: 09/11/2003
• Est. Priority Date: 03/24/2000
• Status: Abandoned Application

• Current Assignee: Reality Commerce Corporation
• Original Assignee: Reality Commerce Corporation
• Inventors: Ao, Yonghui
• Application Number:
  US10/239,415
• Publication Number:
  US 20030172131A1
• Time in Patent Office: Days
• US Class Current: 709/219
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/440227 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/4621 Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/47205 for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6377 directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/643 Communication protocols network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP], H04L65/65
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

A client and server deliver and play subjective video content over the Internet or other network. Frame order, frame rate, and viewing parameters are solely determined by the viewer. A passive streaming protocol supports the operation of the subjective video streaming, in which the server plays a passive role, yielding the control of the entire streaming process to the client system. A scheduler at the client drives the streaming and controls the pace and order of video content downloading. Streaming policies effectively maximize utilization of remote multi-viewpoint image contents shared by multiple on-line viewers.

Claims

1. A method of supporting subjective video at a server, comprising:

receiving a request relating to subjective video content;

accessing a view at will file corresponding to said subjective video content;

in response to said request relating to said subjective video content, providing initial image data relating to an origin processing group of said view at will file;

receiving a subsequent request relating to said subjective video content;

determining, from said subsequent request, a processing group identifier; and

based on said processing group identifier, providing subsequent image data relating to a processing group identified by said processing group identifier;

wherein said initial image data and said subsequent image data comprise coded image data not derived from a three-dimensional model.

2. The method of supporting subjective video at a server as set forth in claim 1, further comprising, after said accessing of said view at will file, obtaining from said view at will file an offset table, wherein said offset table indicates a start of each set of image data relating to each processing group in said view at will file.

3. The method of supporting subjective video at a server as set forth in claim 2, wherein said view at will file comprises:

a file header and processing group code streams;

said file header comprising said offset table;

each of said processing group code streams comprising;

a respective processing group header indicating a processing group, and identifier relating to a control camera in said processing group, and coding parameters; and

a processing group data body, comprising;

a code stream relating to an image provided by said control camera, defining a C-image; and

code streams relating to images provided by each of a plurality of surrounding cameras in said processing group, defining S-images.

4. The method of supporting subjective video at a server as set forth in claim 3, wherein said code streams relating to said C-image and said S-images further comprise a base layer and a set of enhancement layers, said base layer containing information of said image data at a coarse level, and said enhancement layers containing information at finer levels of resolution.

5. A method of supporting subjective video at a. client, comprising:

initiating a streaming process by sending a request relating to subjective video content;

receiving initial image data relating to an origin processing group of said view at will file;

sending a subsequent request relating to a different processing group with respect to said subjective video content;

receiving subsequent image data relating to said different processing group;

wherein said initial image data and said subsequent image data comprise coded image data not derived from a three-dimensional model.

6. The method of supporting subjective video at said client as set forth in claim 5, further comprising:

providing said client with a streaming client and a viewer, said streaming client including a streaming scheduler, said viewer including a viewer controller, a display buffer, an end-user interface, a cache, and an image decoder;

providing said client with a viewpoint map, shared by said streaming client and said viewer;

receiving, in accordance with said initial image data, session description information; and

initializing said viewpoint map based on said session description information;

wherein;

said sending of said initial request activates said streaming scheduler;

said sending of said subsequent request is performed by said streaming scheduler;

said streaming scheduler identifies a selected processing group identifier based on user input;

said streaming scheduler updates said viewpoint map based on said received image data to indicate local availability with respect to image data on a processing group basis;

under control of said viewer controller;

said cache receives said image data in a compressed form;

said image decoder decodes said image data in said compressed form to provide decoded image data; and

said end-user interface receives said coded image data from said display buffer for display.

7. The method of supporting subjective video at said client as set forth in claim 6, wherein said viewer further comprises a geometric functions module for supporting user manipulation operations.

8. The method of supporting subjective video at said client as set forth in claim. 7, wherein said user manipulation operations include zoom, rotation, and revolution.

9. The method of supporting subjective video at said client as set forth in claim 8, wherein said rotation is performed as a solely local function, using a two-dimensional image plane, at said client without support from a server.

10. The method of supporting subjective video at said client as set forth in claim 8, wherein said zoom is performed as a function using support from said client and a remote server using resolution re-scaling operations.

11. The method of supporting subjective video at said client as set forth in claim 5, wherein said steps of sending said subsequent request and receiving said subsequent image data are performed in a synchronous manner.

12. The method of supporting subjective video at said client as set forth in claim 5, wherein said steps of sending said subsequent request and receiving said subsequent image data are performed in an asynchronous manner.

13. The method of supporting subjective video at said client as set forth in claim 6, wherein said streaming scheduler streams image data according to a wave-front model.

14. The method of supporting subjective video at said client as set forth in claim 13, wherein said wave-front model comprises:

when a change of viewpoint is not indicated by a user, s said streaming scheduler requests image data relating to processing groups in proximity to a present processing group, and

when a change of viewpoint is indicated by said user, said streaming scheduler requests image data relating to a processing group at said viewpoint and also processing groups in proximity thereto.

15. The method of supporting subjective video at said client as set forth in claim 13, wherein said wave-front model comprises arranging the order of image download based on the priority of a download task being inversely proportional to a distance between a current viewpoint and a viewpoint where said download task is defined.

16. The method of supporting subjective video at said client as set forth in claim 6, wherein said streaming scheduler streams image data according to a resolution scalability scheduling policy.

17. The method of supporting subjective video at said client as set forth in claim. 16, wherein said resolution scalability scheduling policy comprises:

determining a bandwidth of a local communication connection;

requesting one or more enhancement layers based on said bandwidth determination.

18. The method of supporting subjective video at said client as set forth in claim 16, wherein said resolution scalability scheduling policy comprises initially downloading only a base layer of said image data relating to a given viewpoint, monitoring user interaction to determine whether said given viewpoint is revisited, and, when said monitoring indicates that said given viewpoint is revisited, downloading one or more enhancement layers.

19. The method of supporting subjective video at said client as set forth in claim 8, wherein, in response to an indication of said revolution operation, said streaming scheduler streams image data by skipping processing groups in accordance with an indicated speed of rotation.

20. The method of supporting subjective video at said client as set forth in claim 6, further comprising storing downloaded compressed image data locally and, in response to a request for re-displaying said locally stored downloaded compressed image data, performing the steps of loading said locally stored downloaded compressed image data into said cache;

decoding said locally stored downloaded compressed image data with said image decoder to provide said decoded image data; and

providing said decoded image data to said end-user interface via said display buffer for display.

21. The method of supporting subjective video at said client as set forth in claim 5, wherein said image data is panoramic image data.

22. The method of supporting subjective video at said client as set forth in claim 5, wherein said image data is multi-viewpoint image data.

23. The method of supporting subjective video at said client as set forth in claim 5, wherein said viewer and said streaming client are implemented as plug-ins to a browser.

24. An interactive multi-viewpoint subjective video streaming system, comprising a client and a passive streaming server, said client providing to said server selection commands selecting from a plurality of viewpoints relating to a given scene, said server responding to said commands of said client by providing to said client corresponding image data for said selected one of said plurality of viewpoints.