Dynamic quality adjustment based on changing streaming constraints

• US 7,058,721 B1
• Filed: 08/03/1998
• Issued: 06/06/2006
• Est. Priority Date: 07/14/1995
• Status: Expired due to Term

• Current Assignee: Alcatel-Lucent SA (Nokia Corporation)
• Original Assignee: Broadband Royalty Corporation (CommScope Holding Company, Inc.), Thirdspace Living Ltd. (Nokia Corporation)
• Inventors: Ellison, Lawrence J., Pawson, David J., Porter, Mark A., Bailey, William E.
• Primary Examiner(s): Dinh, Dung C.
• Application Number:
  US09/128,244
• Time in Patent Office: 2,864 Days
• Field of Search: 709/227, 709/231, 709/232, 709/246, 345/327, 345/328, 348/18, 348/13, 348/399, 725/143, 725/145, 725 88- 92, 715/522
• US Class Current: 709/231
• CPC Class Codes: G11B 2220/20 Disc-shaped record carriers
• CPC Class Codes: G11B 2220/415 Redundant array of inexpensive disks [RAID] systems
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 21/232 Content retrieval operation locally within server, e.g. reading video streams from disk arrays storage management G06F3/0604; details of querying and searching of video data from a database G06F16/739
• CPC Class Codes: H04N 21/23805 Controlling the feeding rate to the network, e.g. by controlling the video pump
• CPC Class Codes: H04N 21/2387 Stream processing in response to a playback request from an end-user, e.g. for trick-play
• CPC Class Codes: H04N 21/2393 involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/44004 : involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/47202 : for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8455 : involving pointers to the content, e.g. pointers to the I-frames of the video stream
• CPC Class Codes: H04N 21/8547 : involving timestamps for synchronizing content
• CPC Class Codes: H04N 21/8549 : Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 7/17336 : Handling of requests in head-ends

Abstract

A method and apparatus for dynamic quality adjustment based on changing streaming constraints is provided. According to one aspect of the present invention, a video stream is sent to a client according to a set of streaming constraints. At least a subset of the video information in the video stream is sent from a first source. Next, a signal is received indicating a relaxation of streaming constraints corresponding to the video stream. In one embodiment, the signal is a freeze frame signal. In another embodiment, the signal is a slow motion signal. In response to the signal, a set of improved quality video information from a second source is accessed and sent to the client. According to one embodiment, the set of improved quality video information comprises a still image. According to another embodiment, the set of improved quality video information comprises a set of preprocessed video information ready to be streamed. As a result of the techniques described herein, an improved quality visual image is available for presentation on the client and, consequently, when a viewer requests a presentation rate that reduces the streaming constraints on a video streaming service, the improved quality video information may be sent using the freed-up portion of the bandwidth previously allocated to the client.

Claims

1. A method of manipulating digital video data comprising:

accessing digital audio-visual data, representing an audio-visual work and including data for a plurality of video frames;

determining a start position for frame data representing at least two of the plurality of frames, wherein said start position of a frame is a location of a first byte of the frame;

generating tag data including data representing the start position and other frame related data for at least two of the plurality of frames; and

storing the tag data separate from the digital audio-visual data.

2. The method of claim 1, wherein the tag data includes a time value for each frame.

3. The method of claim 1, wherein the digital audio-visual data further includes non-video data and the tag data includes data referencing the non-video data.

4. The method of claim 1, wherein the other frame related data includes data representing an end position for each frame.

5. The method of claim 1, wherein the digital audio-visual data represents different types of frames and the other frame related data includes data indicating a frame type for each of the video frames.

6. The method of claim 5, wherein the digital audio-visual data is MPEG data.

7. The method of claim 6, wherein the tag data includes, for each video frame, state data representing a state of one or more state machines.

8. The method of claim 1, wherein the tag data is generated as the audio-visual work is displayed.

9. The method of claim 8, wherein the tag data is saved as a separate file.

10. The method of claim 1, wherein the digital audio-visual data is stored in multiple storages.

11. The method of claim 1, wherein the digital audio-visual data is stored remote from where it is displayed.

12. The method of claim 11, wherein the digital audio-visual data is communicated over a network to the location where it is displayed.

13. The method of claim 1, further comprising:

displaying the audio-visual work from a first position in the digital audio-visual data;

receiving a control signal while displaying;

inspecting the tag data;

determining a target position based on the inspected tag data;

ceasing to perform the audio-visual work from the first position;

accessing the target position in the digital representation of the audio-visual work; and

performing the audio-visual work from the accessed target position.

14. The method of claim 13, wherein the control signal indicates a determinable target position, different from the first position, in the digital data.

15. The method of claim 13, wherein determining the target position includes reading the tag data to determine a start position of a frame for the target.

16. The method of claim 13, wherein determining the target position includes determining a target frame based on the inspected tag data and determining the target position to be a position, within the digital audio-visual data, corresponding to the target frame.

17. The method of claim 13, wherein the control signal indicates a determinable period of time and the target position is determined based on the first position and the determinable period of time.

18. The method of claim 17, wherein inspecting the tag data includes inspecting a time value and determining the target position is based upon the inspected time value and the determinable period of time.

19. The method of claim 13, further comprising communicating prefix data prior to communicating data from the target position.

20. The method of claim 13, wherein the digital audio-visual data represents different types of frames, the other frame related data includes data indicating a frame type for each the video frame and determining the target position is based upon the frame type of the video frames at or near the target position.

21. The method of claim 1, further comprising:

displaying the audio-visual work at a first video-frame display rate;

receiving a control signal, while displaying, indicating that the audio-visual work is to be displayed at a second video-frame display rate different from the first video-frame display rate; and

displaying the audio-visual work at the second video-frame display rate.

22. The method of claim 21, wherein the second video-frame display rate is greater than the first video-frame display rate.

23. The method of claim 22, wherein frames are selected for display at the second video-frame display rate based on video frame types.

24. The method of claim 21, wherein the second video-frame display rate causes the display of the audio-visual data to be in reverse.

25. The method of claim 21, further comprising selecting a selected set of video frames from the audio-visual work based on the second video-frame display rate and sizes of video frame data that correspond to said video frames.

26. The method of claim 25, wherein said selecting the selected set of video frames includes repeatedly performing the steps of:

determining a bit budget; and

determining a size of the frame data that corresponds to a current frame and if the size of the frame data that corresponds to the current frame exceeds said bit budget, then not selecting said current frame as a video frame in said selected set of video frames and if the size of the frame data that corresponds to the current frame does not exceed said bit budget, then selecting said current frame as a video frame in said selected set of video frames.

27. The method of claim 26, wherein the bit budget is based on a first time value associated with a most recently selected video frame, a second time value associated with the current frame, said second presentation rate and a data transfer rate.

28. The method of claim 21, further comprising selecting a selected set of video frames from the audio-visual work based on the second video-frame display rate and on video frame types.

29. The method of claim 28, wherein said sequence of video frame data includes at least one type of video frame data from which said corresponding video frame can be constructed without reference to any other video frame data, and at least one type of video frame data from which said corresponding video frame cannot be constructed without reference to any other video frame data.

30. The method of claim 29, wherein said selecting comprises skipping a frame that requires information that has already been skipped.

31. The method of claim 21, further comprising selecting a selected set of video frames from the audio-visual work and inserting prefix data prior to each selected video frame.

32. The method of claim 1, further comprising:

displaying the audio-visual work in accordance with a set of streaming constraints;

receiving a signal indicating a relaxation of said streaming constraints;

in response to said signal, accessing a set of improved quality information, said improved quality information comprising an improved quality version of at least a subset of the information in said audio-visual work; and

displaying at least a subset of the improved quality information.

33. The method of claim 32, wherein said accessing the set of improved quality information comprises:

determining a first reference point in the audio-visual work;

correlating the first reference point with a second reference point in the set of improved quality information; and

retrieving the subset of the improved quality information based on said second reference point.

34. The method of claim 32, wherein the set of improved quality information comprises a still image.

35. The method of claim 34, wherein the still image takes the form of an image file selected from the group consisting of a JPEG file, a GIF file, a BMP file, a TIFF file, a PIC file, a MAC file and a PCD file.

36. The method of claim 32, wherein the set of improved quality information comprises preprocessed audio-visual information ready to be streamed.

37. The method of claim 32, wherein the signal indicates that information is to be displayed at a slower presentation rate.

38. The method of claim 37, wherein said displaying at least a subset of the set of improved quality information comprises displaying a plurality of still images.

39. The method of claim 37, further comprising sending the subset of the set of improved quality information to a client at an appropriate streaming rate to accommodate the slower presentation rate.

40. The method of claim 32, wherein the signal indicates that information display is to be paused.

41. An audio-visual information delivery system for managing the display of an audio-visual work comprising:

a source of digital audio-visual data representing of an audio-visual work for display, the digital data including frame data representing a plurality of the video frames;

a tag data generator configured to generate tag data representing a start position and other frame related data for at least two of the plurality of frames, wherein said start position of a frame is a location of a first byte of the frame; and

a storage for the tag data, separate from the digital audio-visual data source.

42. The system of claim 41, wherein the tag data includes a time value for each frame.

43. The system of claim 41, wherein the digital audio-visual data further includes non-video data and the other tag data includes data referencing the non-video data.

44. The system of claim 43, wherein the other frame related data includes data representing an end position for each frame.

45. The system of claim 43, wherein the digital audio-visual data represents different types of frames and the other frame related data includes data indicating a frame type for each video frame.

46. The system of claim 45, wherein the digital audio-visual data is MPEG data.

47. The system of claim 46, wherein the tag data includes, for each video frame, state data representing a state of one or more state machines.

48. The system of claim 43, wherein the tag data is generated as the audio-visual work is displayed.

49. The system of claim 48, wherein the tag data is saved as a separate file.

50. The system of claim 43, wherein the digital audio-visual data source includes multiple separate storages.

51. The system of claim 43, wherein the digital audio-visual data source includes storage remote from where the audio-visual work is displayed.

52. The system of claim 51, wherein the digital audio-visual data is communicated over a network from the audio-visual storage to the location where it is displayed.

53. The system of claim 41, wherein the system is configured to:

display the audio-visual work from a first position in the digital audio-visual data;

receive a control signal;

inspect the tag data in response to the received signal;

determine the target position based on the inspected tag data;

cease performing the audio-visual work from the first position;

access the target position in the digital representation of the audio-visual work; and

continue performing the audio-visual work from the accessed target position.

54. The system of claim 53, wherein the control signal indicates a determinable target position, different from the first position, in the digital data.

55. The system of claim 53, wherein determining the target position includes reading the tag data to determine a start position of a frame for the target.

56. The system of claim 53, wherein the target position is determined by determining a target frame based on the inspected tag data and determining the target position to be a position, within the digital audio-visual data, corresponding to the target frame.

57. The system of claim 56, wherein the control signal indicates a determinable period of time and the target position is determined based on the first position and the determinable period of time.

58. The system of claim 57, wherein a time value in the tag data is inspected and the target position determined based upon the inspected time value and the determinable period of time.

59. The system of claim 58, wherein prefix data is inserted prior to data from the target position.

60. The system of claim 56, wherein the digital audio-visual data represents different types of frames, the other frame related data includes data indicating a frame type for each of the video frames and the system is configured to determine the target position based upon the frame type of the video frames at or near the target position.

61. The system of claim 41, wherein when the control signal is received when the audio-visual work is being displayed at a first video-frame display rate, the system is configured to cause the display of the audio-visual work at a second video-frame display rate different from the first video-frame display rate.

62. The system of claim 61, wherein the second video-frame display rate is greater than the first video-frame display rate.

63. The system of claim 62, wherein the system selects frames to display at the second video-frame display rate based on video frame types.

64. The system of claim 61, wherein the system causes the display of the audio-visual data to be in reverse.

65. The system of claim 61, wherein a selected set of video frames are selected from the audio-visual work based on the second video-frame display rate and sizes of video frame data that correspond to said video frames.

66. The system of claim 65, wherein the selected set of video frames are selected by the system repeatedly performing the steps of

determining a bit budget;

and

determining a size of the frame data that corresponds to a current frame and if the size of the frame data that corresponds to the current frame exceeds said bit budget, then not selecting said current frame as a video frame in said selected set of video frames and if the size of the frame data that corresponds to the current frame does not exceed said bit budget, then selecting said current frame as a video frame in said selected set of video frames.

67. The system of claim 66, wherein the bit budget is based on a first time value associated with a most recently selected video frame, a second time value associated with the current frame, said second presentation rate and a data transfer rate.

68. The system of claim 61, wherein a selected set of video frames are selected from the audio-visual work based on the second video-frame display rate and video frame types.

69. The system of claim 68, wherein said sequence of video frame data includes at least one type of video frame data from which said corresponding video frame can be constructed without reference to any other video frame data, and at least one type of video frame data from which said corresponding video frame cannot be constructed without reference to any other video frame data.

70. The system of claim 69, wherein a frame is skipped that requires information that has already been skipped.

71. The system of claim 61, wherein a selected set of video frames are selected from the audio-visual work and prefix data is inserted prior to each selected video frame.

72. The system of claim 41, wherein the system is configured to:

display the audio-visual work in accordance with a set of streaming constraints;

receive a signal indicating a relaxation of said streaming constraints, wherein in response to the signal, the system accesses a set of improved quality information, said improved quality information comprising an improved quality version of at least a subset of the information in said audio-visual work; and

display at least a subset of the improved quality information.

73. The system of claim 72, wherein the system accesses the set of improved quality information by determining a first reference point in the audio-visual work, correlating the first reference point with a second reference point in the set of improved quality information and retrieving the subset of the improved quality information based on said second reference point.

74. The system of claim 72, wherein the set of improved quality information comprises a still image.

75. The system of claim 74, wherein the still image takes the form of an image file selected from the group consisting of a JPEG file, a GIF file, a BMP file, a TIFF file, a PIC file, a MAC file and a PCD file.

76. The system of claim 72, wherein the set of improved quality information comprises preprocessed audio-visual information ready to be streamed.

77. The system of claim 72, wherein the signal indicates that information is to be displayed at a slower presentation rate.

78. The system of claim 77, wherein at least a subset of the set of improved quality information are displayed by displaying a plurality of still images.

79. The system of claim 77, wherein the subset of the set of improved quality information are sent to a client at an appropriate streaming rate to accommodate the slower presentation rate.

80. The system of claim 72, wherein the signal indicates that information display is to be paused.