METHOD AND SYSTEM FOR SMART ADAPTIVE VIDEO STREAMING DRIVEN BY PERCEPTUAL QUALITY-OF-EXPERIENCE ESTIMATIONS

• US 20180041788A1
• Filed: 02/08/2016
• Published: 02/08/2018
• Est. Priority Date: 02/07/2015
• Status: Active Grant

• Current Assignee: IMAX Corporation
• Original Assignee: SSIMWave Inc. (IMAX Corporation)
• Inventors: WANG, Zhou, ZENG, Kai, REHMAN, Abdul
• Granted Patent: US 10,863,217 B2
• Time in Patent Office: Days
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 19/102 characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23805 Controlling the feeding rate to the network, e.g. by controlling the video pump
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

The present invention is a system or method that facilitates smart decision makings at the client side for adaptive video streaming over a video delivery network by making use of perceptual video quality-of-experience predictions performed during the video preparation stage, at the video hosting or cache server side, or inside the video delivery network, and then transmitted to the client. Compared with prior art approaches of adaptive bitrate video streaming, the present invention can result in one or more of the following benefits: 1) save the overall bandwidth for the delivery of the video content without scarifying the client users'"'"' quality-of-experience; 2) create better overall visual quality-of-experience of the client users; 3) create smoother visual quality-of-experience of the client users; and 4) reduce the probability of rebuffering or stalling events at the client user side.

Claims

1. A method for adaptive video streaming over video delivery networks, comprising:

(a) Creating multiple video streams of different bitrates and resolutions from the same video source content, and dividing each of the video streams into time segments;

(b) Performing quality-of-experience predictions of the video streams during the video preparation stage, at the video hosting site, and/or inside the video delivery network;

(c) Transmitting the quality-of-experience prediction parameters prior to or together with the video streams to the receiver client site;

(d) Using the received quality-of-experience prediction parameters and client side network, device and viewing environment information to estimate the actual user quality-of-experience (“

QOE”

) at the receiver client side to obtain a QoE estimate; and

(e) Using the user QoE estimate, at the receiver client side, to select a video stream from the multiple video streams and to request the next time segments of the selected video stream.

3. The method of claim 1, further comprising performing human subjective quality-of-experience measurement during the video preparation stage, at the video hosting site, or inside the video delivery network, and use the subjective ratings as the quality-of-experience predictors.

4. The method of claim 1, further comprising using computational full-reference, reduced-reference, and/or no-reference objective video quality assessment models such as PSNR, SSIM, MS-SSIM, VQM, MOVIE and SSIMplus, as perceptual quality-of-experience predictors.

5. The method of claim 1, further comprising using full-reference and/or reduced-reference objective perceptual models that produce parameters that are able to compare video quality across different spatial and/or temporal resolutions as the quality-of-experience predictors.

6. The method of claim 1, further comprising using objective perceptual video quality models that produce parameters that predict perceptual quality-of-experience dependent on the type and settings of the viewing device, the resolution of the playback window on the viewing device, and/or the viewing conditions of the video at the client site.

7. The method of claim 1, further comprising transmitting the quality-of-experience prediction parameters as metadata prior to the transmission of the video streams or together with the transmission of video streams.

8. The method of claim 1, further comprising transmitting the quality-of-experience prediction parameters by embedding them as watermarks or hidden messages into the video streams.

9. The method of claim 1, further comprising creating a matrix of viewer quality-of-experience estimation for each segment of each video stream at the client site based on the viewer quality-of-experience method of claim 2, and then use the matrix in the streaming decision making step on the selection of the next segments of video.

10. The method of claim 1, further comprising a streaming decision making step on the selection of the next segment of the video at the client site that combines quality-of-experience estimation with other available information, including one or more of the bitrates of video streams, the resolutions of the video streams, the available bandwidth of the network, and the decoding speed, display speed, buffer size and power of the receiver device.

11. The method of claim 1, further comprising a streaming decision making step on the selection of the next segment of the video at the client site that picks the maximal quality-of-experience video stream, under the constraints of video bitrate, network bandwidth, decoding speed, display speed, buffer size and device power.

12. The method of claim 1, further comprising a streaming decision making step on the selection of the next segment of the video at the client site to save bandwidth, to reduce the probability of rebuffering, to improve the overall quality-of-experience, and/or to maintain smoothness of quality-of-experience, by rejecting to switch to an affordable higher bitrate and/or higher resolution stream, when without such switching, the quality-of-experience maintains at or above a pre-determined target threshold level.

13. The method of claim 1, further comprising a streaming decision making step on the selection of the next segment of the video at the client site to save bandwidth, to reduce the probability of rebuffering, to improve the overall quality-of-experience, and/or to maintain smoothness of quality-of-experience, by rejecting to switch to an affordable higher bitrate and/or higher resolution stream when such switching results in quality-of-experience increases lower than a threshold value.

14. The method of claim 1, further comprising a streaming decision making step on the selection of the next segment of the video at the client site to save bandwidth, to reduce the probability of rebuffering, to improve the overall quality-of-experience, and/or to maintain smoothness of quality-of-experience, by switching to a lower bitrate and/or lower resolution stream, with or without seeing a drop in network bandwidth or buffer size, when such switching results in quality-of-experience drops lower than a threshold value, and/or when with such switching, the quality-of-experience maintains at or above a pre-determined target threshold quality-of-experience level.

15. The method of claim 1, further comprising a streaming decision making step on the selection of the next segment of the video at the client site to save bandwidth, to reduce the probability of rebuffering, to improve the overall quality-of-experience, and/or to maintain smoothness of quality-of-experience, by switching to a lower bitrate and/or lower resolution stream, with or without seeing a drop in network bandwidth or buffer size, when foreseeing future video segments that need higher than the current bitrate to maintain the same level of quality-of-experience.

16. The method of claim 1, further comprising a streaming decision making step on the selection of the next segment of the video at the client site to maintain the current level and smoothness of quality-of-experience by switching to a stream of higher bitrate and/or higher resolution stream, with or without seeing an increase in network bandwidth or buffer size, when without such switching, the quality-of-experience drops more than a threshold value, and when the absolute difference in quality-of-experience between the higher bitrate stream and the current stream at the next segment is lower than another threshold.

17. The method of claim 1, further comprising a streaming decision making step on the joint selections of a sequence of the next multiple segments of the video at the client site by performing a dynamic programming optimization such as the Viterbi'"'"'s algorithm to decide on the best path that maximizes the average quality and/or smoothness of quality-of-experience.

2. A method that makes a quality-of-experience estimation at the client site statically by using the quality-of-experience prediction parameters received from the network, or dynamically by combining quality-of-experience prediction parameters received from the network with one or multiple instant network and receiver conditions, wherein the conditions include erroneous transmission and/or decoding, initial buffering and rebuferring, pixel resolution of viewing device, physical size of viewing device, video frame pixel resolution on device, video temporal resolution, video playback speed on device, viewing environment condition, user preference, user vision condition, or user expectation.

18-33. -33. (canceled)