Method and system for complexity adaptive streaming

• US 9,967,302 B2
• Filed: 11/11/2013
• Issued: 05/08/2018
• Est. Priority Date: 11/14/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lim, Youngkwon, Ma, Zhan, Bhat, Kong Posh, Fernandes, Felix C. A.
• Primary Examiner(s): Nano, Sargon
• Application Number:
  US14/076,998
• Publication Number:
  US 20140136727A1
• Time in Patent Office: 1,639 Days
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 65/80 Responding to QoS

Abstract

A method includes calculating a complexity value for each segment or version of multimedia content. The multimedia content has multiple versions associated with common content and different encoding parameters, and each version is divided into segments. The method also includes adaptively streaming the multimedia content to a client device based on at least some of the calculated complexity values. The complexity value for each segment or version is calculated by summing multiple operation complexity values defining complexities of operations for decoding the segment or version. The operation complexity values could include values for a bit parsing operation, a buffer initialization operation, an intra-prediction operation, an inverse quantization and transform operation, a reference block fetch operation, a fractional interpolation operation, and in-loop deblocking operation, a sample adaptive offset correction operation, an adaptive loop filtering operation, and a de-ringing filtering operation.

Claims

1. A method comprising:

calculating a complexity value for each segment or version of multimedia content, the multimedia content having multiple versions associated with common content and different encoding parameters, each version divided into segments; and

adaptively streaming the multimedia content to a client device based on at least some of the calculated complexity values, wherein adaptively streaming the multimedia content comprises;

streaming a first version of the multimedia content to the client device; and

switching to streaming a second version of the multimedia content to the client device based on a power status of the client device;

wherein the complexity value for each segment or version is calculated by summing multiple operation complexity values defining complexities of operations for decoding the segment or version.

2. The method of claim 1, wherein:

the versions of the multimedia content include different versions associated with different encoding standards; and

the operations for decoding vary between at least some of the encoding standards.

3. The method of claim 2, wherein the different versions of the multimedia content have different bitrates and require different computational powers to decode.

4. The method of claim 1, wherein:

all versions are associated with bit parsing, buffer initialization, inverse quantization and transform, and reference block fetch operations;

a first subset of versions are also associated with intra-prediction, fractional interpolation, and in-loop deblocking operations;

a second subset of versions are also associated with intra-prediction, fractional interpolation, in-loop deblocking, and sample adaptive offset correction operations; and

a third subset of versions are also associated with intra-prediction, fractional interpolation, in-loop deblocking, sample adaptive offset correction, adaptive loop filtering, and de-ringing filtering operations.

5. The method of claim 4, wherein the complexity value for each segment or version is calculated as:

C_total=C_{bit_parsing}+C_{buf_init}+C_intra+C_itrans+C_{ref_fetch}+C_intp+C_dblk+C_sao+C_alf+C_drf 

 

(1)

where;

C_total represents the complexity value for the segment or version;

C_{bit_parsing} represents the operation complexity value of the bit parsing operation;

C_{buf_init} represents the operation complexity value of the buffer initialization operation;

C_intra represents the operation complexity value of the intra-prediction operation;

C_itrans represents the operation complexity value of the inverse quantization and transform operation;

C_{ref_fetch} represents the operation complexity value of the reference block fetch operation;

C_intp represents the operation complexity value of the fractional interpolation operation;

C_dblk represents the operation complexity value of the in-loop deblocking operation;

C_sao represents the operation complexity value of the sample adaptive offset correction operation;

C_alf represents the operation complexity value of the adaptive loop filtering operation; and

C_drf represents the operation complexity value of the de-ringing filtering operation.

6. The method of claim 5, wherein the operation complexity values are calculated as:

C_{bit_parsing}=k_bit×

n_bit

C_{buf_init}=k_{buf_init}×

n_total

C_intra=k_intra×

n_intra

C_itrans=k_itrans×

(n_total−

n_zero)

C_{ref_fetch}=k_{ref_fetch}×

n_total

C_intp=k_intp×

n_intp

C_dblk=k_dblk×

n_dblk

C_sao=k_sao×

n_sao

C_alf=k_alf×

n_alf

C_drf=k_drf×

n_drf

where;

n_bit represents a number of bits;

n_total represents a number of blocks;

n_intra represents an intra-block number;

n_zero represents a number of zero blocks

n_intp represents a number of interpolations;

n_dblk represents a number of pixels requiring deblocking;

n_sao represents a number of pixels requiring sample adaptive offset correction;

n_alf represents a number of blocks requiring adaptive loop filtering;

n_drf represents a number of blocks requiring de-ringing filtering; and

k_bit, k_{buf_init}, k_intra, k_itrans, k_{ref_fetch}, k_intp, k_dblk, k_sao, k_alf, and k_drf denote unit operation complexity values.

7. The method of claim 1, wherein:

streaming the first version comprises streaming the first version in response to a first request from the client device; and

switching to streaming the second version comprises selecting the second version in response to a second request from the client device.

8. The method of claim 1, further comprising:

providing a list of the different versions of the multimedia content to the client device.

9. An apparatus comprising:

at least one network interface configured to communicate with a client device over a network; and

at least one processing device configured to;

calculate a complexity value for each segment or version of multimedia content, the multimedia content having multiple versions associated with common content and different encoding parameters, each version divided into segments; and

adaptively stream the multimedia content via the at least one network interface to the client device based on at least some of the calculated complexity values, wherein the at least one processing device is configured to adaptively stream the multimedia content by;

streaming a first version of the multimedia content to the client device; and

switching to streaming a second version of the multimedia content to the client device based on a power status of the client device;

wherein the at least one processing device is configured to calculate the complexity value for each segment or version by summing multiple operation complexity values defining complexities of operations for decoding the segment or version.

10. The apparatus of claim 9, wherein:

the versions of the multimedia content include different versions associated with different encoding standards; and

the operations for decoding vary between at least some of the encoding standards.

11. The apparatus of claim 9, wherein:

all versions are associated with bit parsing, buffer initialization, inverse quantization and transform, and reference block fetch operations;

a first subset of versions are also associated with intra-prediction, fractional interpolation, and in-loop deblocking operations;

a second subset of versions are also associated with intra-prediction, fractional interpolation, in-loop deblocking, and sample adaptive offset correction operations; and

a third subset of versions are also associated with intra-prediction, fractional interpolation, in-loop deblocking, sample adaptive offset correction, adaptive loop filtering, and de-ringing filtering operations.

12. The apparatus of claim 11, wherein the at least one processing device is configured to calculate the complexity value for each segment or version as:

C_total=C_{bit_parsing}+C_{buf_init}+C_intra+C_itrans+C_{ref_fetch}+C_intp+C_dblk+C_sao+C_alf+C_drf

where;

C_total represents the complexity value for the segment or version;

C_{bit_parsing} represents the operation complexity value of the bit parsing operation;

C_{buf_init} represents the operation complexity value of the buffer initialization operation;

C_intra represents the operation complexity value of the intra-prediction operation;

C_itrans represents the operation complexity value of the inverse quantization and transform operation;

C_{ref_fetch} represents the operation complexity value of the reference block fetch operation;

C_intp represents the operation complexity value of the fractional interpolation operation;

C_dblk represents the operation complexity value of the in-loop deblocking operation;

C_sao represents the operation complexity value of the sample adaptive offset correction operation;

C_alf represents the operation complexity value of the adaptive loop filtering operation; and

C_drf represents the operation complexity value of the de-ringing filtering operation.

13. The apparatus of claim 12, wherein the at least one processing device is configured to calculate the operation complexity values as:

C_{bit_parsing}=k_bit×

n_bit

C_{buf_init}=k_{buf_init}×

n_total

C_intra=k_intra×

n_intra

C_itrans=k_itrans×

(n_total−

n_zero)

C_{ref_fetch}=k_{ref_fetch}×

n_total

C_intp=k_intp×

n_intp

C_dblk=k_dblk×

n_dblk

C_sao=k_sao×

n_sao

C_alf=k_alf×

n_alf

C_drf=k_drf×

n_drf

where;

n_bit represents a number of bits;

n_total represents a number of blocks;

n_intra represents an intra-block number;

n_zero represents a number of zero blocks

n_intp represents a number of interpolations;

n_dblk represents a number of pixels requiring deblocking;

n_sao represents a number of pixels requiring sample adaptive offset correction;

n_alf represents a number of blocks requiring adaptive loop filtering;

n_drf represents a number of blocks requiring de-ringing filtering; and

k_bit, k_{buf_init}, k_intra, k_itrans, k_{ref_fetch}, k_intp, k_dblk, k_sao, k_alf, and k_drf denote unit operation complexity values.

14. The apparatus of claim 9, wherein the at least one processing device is further configured to provide a list of the different versions of the multimedia content to the client device.

15. A method comprising:

receiving a list of available versions of multimedia content at a client device, the multimedia content having multiple versions associated with common content and different encoding parameters, each version divided into segments;

identifying whether the client device has a specified power status;

when the client device has the specified power status, selecting or switching to a segment or version that consumes less power for decoding; and

receiving and rendering the segment or version of multimedia content;

wherein the selecting or switching comprises selecting or switching to a segment or version having a lower complexity value, the complexity value comprising a sum of multiple operation complexity values defining complexities of operations for decoding the segment or version.

16. The method of claim 15, wherein the selecting or switching occurs without discontinuing use of a streaming service.

17. The method of claim 15, wherein the complexity value for each segment or version is defined as:

C_total=C_{bit_parsing}+C_{buf_init}+C_intra+C_itrans+C_{ref_fetch}+C_intp+C_dblk+C_sao+C_alf+C_drf

where;

C_total represents the complexity value for the segment or version;

C_{bit_parsing} represents the operation complexity value of a bit parsing operation;

C_{buf_init} represents the operation complexity value of a buffer initialization operation;

C_intra represents the operation complexity value of an intra-prediction operation;

C_itrans represents the operation complexity value of an inverse quantization and transform operation;

C_{ref_fetch} represents the operation complexity value of a reference block fetch operation;

C_intp represents the operation complexity value of a fractional interpolation operation;

C_dblk represents the operation complexity value of an in-loop deblocking operation;

C_sao represents the operation complexity value of a sample adaptive offset correction operation;

C_alf represents the operation complexity value of an adaptive loop filtering operation; and

C_drf represents the operation complexity value of a de-ringing filtering operation.

18. The method of claim 17, wherein the operation complexity values are defined as:

C_{bit_parsing}=k_bit×

n_bit

C_{buf_init}=k_{buf_init}×

n_total

C_intra=k_intra×

n_intra

C_itrans=k_itrans×

(n_total−

n_zero)

C_{ref_fetch}=k_{ref_fetch}×

n_total

C_intp=k_intp×

n_intp

C_dblk=k_dblk×

n_dblk

C_sao=k_sao×

n_sao

C_alf=k_alf×

n_alf

C_drf=k_drf×

n_drf

where;

n_bit represents a number of bits;

n_total represents a number of blocks;

n_intra represents an intra-block number;

n_zero represents a number of zero blocks

n_intp represents a number of interpolations;

n_dblk represents a number of pixels requiring deblocking;

n_sao represents a number of pixels requiring sample adaptive offset correction;

n_alf represents a number of blocks requiring adaptive loop filtering;

n_drf represents a number of blocks requiring de-ringing filtering; and

k_bit, k_{buf_init}, k_intra, k_itrans, k_{ref_fetch}, k_intp, k_dblk, k_sao, k_alf, and k_drf denote unit operation complexity values.