Distributed Architecture for Encoding and Delivering Video Content

• US 20130343450A1
• Filed: 06/11/2013
• Published: 12/26/2013
• Est. Priority Date: 06/12/2012
• Status: Active Grant

• Current Assignee: Coherent Logix, Incorporated
• Original Assignee: Coherent Logix, Incorporated
• Inventors: Solka, Michael B., Doerr, Michael B., Dobbs, Carl S., Bruns, Michael W.
• Granted Patent: US 11,483,580 B2
• Time in Patent Office: Days
• US Class Current: 375/240.03
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/19 using optimisation based on Lagrange multipliers
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2353 specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/25808 Management of client data terminal profiles in network data switching protocols H04L67/303
• CPC Class Codes: H04N 21/25825 involving client display capabilities, e.g. screen resolution of a mobile phone optimising the visualisation of content during browsing in the Internet G06F16/9577; processing of terminal status or physical abilities in wireless networks H04W8/22; authentication in wireless network security H04W12/06
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities

Abstract

A split architecture for encoding a video stream. A source encoder may encode a video content stream to obtain an encoded bitstream and a side information stream. The side information stream includes information characterizing rate and/or distortion estimation functions per block of the video content stream. Also, a different set of estimation functions may be included per coding mode. The encoded bitstream and side information stream may be received by a video transcoder, which transcodes the encoded bitstream to a client-requested picture resolution, according to a client-requested video format and bit rate. The side information stream allows the transcoder to efficient and compactly perform rate control for its output bitstream, which is transmitted to the client device. This split architecture may be especially useful to operators of content delivery networks.

Claims

1. A system comprising:

a memory that stores a collection of video content items, wherein each of the video content items includes a corresponding encoded video stream and corresponding side information stream;

one or more video transcoder devices; and

a controller, wherein, in response to each of a plurality of content requests for a given one of the video content items from a respective plurality of user devices, the controller is configured to assign an available one of the one or more video transcoder devices to serve the respective user device, wherein the user devices have respectively different configurations of video processing capability (VPC), wherein each assigned video transcoder device is configured to;

receive the encoded video stream and side information stream of the given video content item; and

transcode the encoded video stream using the side information stream and according to the VPC configuration of the respective user device, in order to obtain a respective target encoded video stream; and

transmit the respective target encoded video stream to the respective user device through a communication medium.

2. The system of claim 1, wherein the encoded video stream of the given video content item is an encoded version of a given source video stream, wherein the side information stream includes metadata that characterizes properties of the given source video stream.

3. The system of claim 2, wherein the metadata includes one or more candidate motion vectors per block of the encoded video stream of the given content item.

4. The system of claim 2, wherein the metadata includes rate modeling data per block of the encoded video stream of the given content item.

5. The system of claim 2, wherein the metadata includes distortion modeling data per block of the encoded video stream of the given content item.

6. The system of claim 1, wherein the communication medium is a wireless transmission medium.

7. The system of claim 6, wherein at least one of the one or more video transcoder devices is coupled to or incorporated as part of a base station of a wireless communication network, wherein one or more of the user devices are configured for wireless communication with the base station.

8. The system of claim 1, wherein the controller is configured to assign a first of the one or more video transcoder devices to different ones of the user devices at different times.

9. The system of claim 1, further comprising:

a source encoder configured to encode source video streams to generate respective ones of the content items, wherein each of the one or more video transcoder devices is more power efficient than the source encoder, and/or more space efficient than the source encoder.

10. The system of claim 1, wherein the side information stream of each video content item includes one or more rate information streams corresponding to one or more respective coding modes, wherein each rate information stream RIS_k of the one or more rate information streams characterizes a corresponding rate estimation function R_k(q) per block of the corresponding encoded video stream, assuming block prediction based on the respective coding mode, wherein q is a quantization step size.

11. The system of claim 1, wherein the side information stream of each video content item includes one or more distortion information streams corresponding to one or more respective coding modes, wherein each distortion information stream DIS_k of the one or more distortion information streams characterizes a corresponding distortion function D_k(q) per block of the encoded video stream of the video content item, assuming block prediction based on the respective coding mode, wherein q is a quantization step size.

12. The system of claim 1, wherein a first of the one or more video transcoder devices is configured to perform said transcoding by:

decoding the encoded video stream to obtain a decoded video stream;

scaling the decoded video stream to a target picture resolution of the respective user device, in order to obtain a scaled video stream; and

encoding the scaled video stream using the side information and according to the VPC configuration of the respective user device, in order to obtain the respective target encoded video stream.

13. The system of claim 12, wherein the controller is configured to:

receive reports from the user device being served by the first video transcoder device, wherein each of the reports includes analytical information from the user device; and

in response to each of the reports, update a target bit rate and/or the target picture resolution used by the first video transcoder device to encode the scaled video stream.

14. The system of claim 13, wherein the analytical information includes information about quality of a link between the first video transcoder device and the user device.

15. The system of claim 14, wherein the controller is configured to decrease or increase the target bit rate and/or the target picture resolution used by the first video transcoder device when the information about link quality indicates that the link quality has decreased or increased, respectively.

16. The system of claim 13, wherein the analytical information includes information about the quality of video recovered from the respective target encoded video stream transmitted by the first video transcoder device.

17. The system of claim 16, wherein the controller is configured to decrease or increase the target bit rate and/or the target picture resolution used by the first video transcoder device when the information about video quality indicates that the video quality has decreased or increased, respectively.

18. The system of claim 12, wherein the controller is configured to:

receive reports from the user device being served by the first video transcoder device, wherein each of the reports includes a corresponding update to the VPC configuration of the user device; and

in response to each of the reports, update the target picture resolution used by the first video transcoder device to encode the scaled video stream.

19. The system of claim 1, wherein the VPC configuration of each user device includes an identification of one or more of:

a video coding format requested by the user device; and

a target picture resolution requested by the user device.

20. The system of claim 1, wherein the VPC configurations of the respective user devices span an M-dimensional configuration space, wherein M is at least two, wherein the M-dimensional configuration space has at least a first dimension corresponding to a choice of video format and a second dimension corresponding to a selection of picture resolution.

21. The system of claim 1, wherein the controller is configured to:

store the target encoded video stream generated by a given one of the one or more video transcoder devices that has been assigned to serve a first of the user devices; and

direct a transmission of the stored target encoded video stream to a second user device in response to detecting that the second user device has a same or similar VPC configuration as the first user device.

22. A method for delivering video content to user devices, the method comprising:

storing a collection of video content items in a memory, wherein each of the video content items includes a corresponding encoded video stream and corresponding side information stream;

in response to each of a plurality of content requests for a given one of the video content items from a respective plurality of remote user devices, assigning an available one of one or more video transcoder devices to serve the respective user device, wherein the user devices have respectively different configurations of video processing capability (VPC);

utilizing each assigned video transcoder device to;

receive the encoded video stream and side information stream of the given video content item;

transcode the encoded video stream using the side information stream and according to the VPC configuration of the respective user device, in order to obtain a respective target encoded video stream; and

transmit the respective target encoded video stream to the respective user device through a communication medium.

23. The method of claim 22, wherein the encoded video stream of the given video content item is an encoded version of a given source video stream, wherein the side information stream includes metadata that characterizes properties of the given source video stream.

24. The method of claim 23, wherein the metadata includes one or more candidate motion vectors per block of the encoded video stream of the given video content item.

25. The method of claim 23 wherein the metadata includes rate modeling data per block of the encoded video stream of the given video content item.

26. The method of claim 23, wherein the metadata includes distortion modeling data per block of the encoded video stream of the given video content item.

27. The method of claim 22, wherein the side information stream of each video content item includes one or more rate information streams corresponding to one or more respective coding modes, wherein each rate information stream RIS_k of the one or more rate information streams characterizes a corresponding rate estimation function R_k(q) per block of the corresponding encoded video stream, assuming block prediction based on the respective coding mode, wherein q is a quantization step size.

28. The method of claim 22, wherein the side information stream of each video content item includes one or more distortion information streams corresponding to one or more respective coding modes, wherein each distortion information stream DIS_k of the one or more distortion information streams characterizes a corresponding distortion function D_k(q) per block of the encoded video stream of the given video content item, assuming block prediction based on the respective coding mode, wherein q is a quantization step size.

29. The method of claim 22, further comprising:

encoding source video streams to generate respective ones of the content items, wherein said encoding the source video stream is performed by a source encoder, wherein each of the video transcoder devices consumes less power than the source encoder, and occupies less space than the source encoder.

30. The method of claim 22, further comprising:

receiving reports from the user device being served by a first of the one or more video transcoder devices, wherein each of the reports includes analytical information from the user device; and

in response to each of the reports, update a target bit rate and/or the target picture resolution used by the first video transcoder device to perform said transcoding of the encoded video stream.

31. The method of claim 30, wherein the analytical information includes information about quality of a link between the first video transcoder device and the user device.

32. The method of claim 31, further comprising:

decreasing or increasing the target bit rate and/or the target picture resolution used by the first video transcoder device when the information about link quality indicates that the link quality has decreased or increased, respectively.

33. The method of claim 30, wherein the analytical information includes information about the quality of video recovered from the respective target encoded video stream transmitted by the first video transcoder device.

34. The system of claim 33, further comprising:

decreasing or increasing the target bit rate and/or the target picture resolution used by the first video transcoder device when the information about video quality indicates that the video quality has decreased or increased, respectively.

35. The method of claim 22, further comprising:

receiving reports from the user device being served by the first video transcoder device, wherein each of the reports includes a corresponding update to the VPC configuration of the user device; and

in response to each of the reports, updating the target picture resolution used by the first video transcoder device to encode the scaled video stream.

36. The method of claim 22, wherein the VPC configuration of each user device includes an identification of a video coding format requested by the user device, wherein said transcoding of the encoded video stream is performed so that the respective target encoded video stream conforms to the requested video coding format.

37. The method of claim 22, wherein the VPC configuration of each user device includes an identification of a target picture resolution requested by the user device, wherein said transcoding of the encoded video stream is performed so that the respective target encoded video stream has the requested target picture resolution.

38. The method of claim 22, wherein the VPC configuration of each user device includes an identification of a target bit rate requested by the user device, wherein said transcoding of the encoded video stream is performed so that the respective target encoded video stream has an average output bit rate approximately equal to the requested target bit rate.

39. The method of claim 22, further comprising:

storing the target encoded video stream generated by a given one of the one or more video transcoder devices that has been assigned to serve a first of the user devices; and

directing a transmission of the stored target encoded video stream to a second user device in response to detecting that the second user device has a same or similar VPC configuration as the first user device.

40. A video encoder comprising:

digital circuitry configured to perform, for each of a plurality of blocks of the input video stream, operations including;

transforming a plurality of prediction residuals that correspond respectively to one or more coding modes in order to obtain one or more respective transform blocks for the one or more respective coding modes;

for each coding mode M_k, processing the respective prediction residual and/or the respective transform block for the coding mode M_k to obtain rate modeling data for the coding mode M_k;

transmission circuitry configured to transmit a side information stream onto a communication medium, wherein the side information stream includes the rate modeling data for each coding mode and for each block.

41. The video encoder of claim 40, wherein the rate modeling data includes data characterizing a rate estimation function R_k(q) for at least one of the one or more coding modes, wherein q represents quantization step size.

42. The video encoder of claim 40, wherein the operations also include:

for each coding mode M_k, generating one or more reconstruction residuals based respectively on one or more quantized versions of the transform block for that coding mode, and generating distortion modeling data for the coding mode based on the one or more reconstruction residuals, wherein the side information stream also includes the distortion modeling data for each coding mode and each block.

43. The video encoder of claim 42, wherein the distortion modeling data includes data characterizing a distortion estimation function D_k(q) for at least one of the one or more coding modes, wherein q represents quantization step size.

44. The video encoder of claim 40, wherein the digital circuitry is further configured to generate an encoded video stream that represents an encoded version of the input video stream, wherein the transmission circuitry is configured to transmit the encoded video stream onto the communication medium.

45. The video encoder of claim 44, wherein said generating the encoded video stream includes operating on at least one of the one or more transform blocks.

46. A video encoder comprising:

digital circuitry configured to encode an input video stream to obtain an encoded video stream, wherein said encoding includes generating a side information stream that characterizes properties of the input video stream; and

transmission circuitry configured to transmit the encoded video stream and the side information stream.

47. The video encoder of claim 46, wherein the side information stream includes data characterizing rate and/or distortion properties of the input video stream.

48. The video encoder of claim 46, wherein the side information stream includes a stream of candidate motion vectors.

49. A video transcoding system comprising:

a decoder configured to receive and decode a first encoded video stream to obtain a decoded video stream;

a scaling unit configured to scale the decoded video stream to a target picture resolution in order to obtain a scaled video stream;

an output encoder configured to receive a side information stream associated with the first encoded video stream, and encode the scaled video stream using the side information stream in order to obtain a second encoded video stream.

50. The video transcoding system of claim 49, wherein the first encoded video stream is an encoded version of a source video stream, wherein the side information stream includes metadata that characterizes properties of the source video stream.

51. The video transcoding system of claim 49, wherein the target picture resolution is lower than a picture resolution implicit in the first encoded video stream.

52. The video transcoding system of claim 49, wherein the side information stream includes N rate information streams corresponding to N respective coding modes, wherein N is greater than or equal to one, wherein each rate information stream RIS_k of the N rate information streams characterizes a corresponding rate estimation function R_k(q) per block of the first encoded video stream assuming block prediction based on the respective coding mode, wherein q is a quantization step size.

53. The video transcoding system of claim 52, wherein N=2 for a given coded picture of the first encoded video stream, wherein the rate information stream RIS_C corresponds to an intra coding mode, wherein the rate information stream RIS₂ corresponds to an inter coding mode.

54. The video transcoding system of claim 52, wherein each rate information stream RIS_k characterizes the rate estimation function R_k(q) for each block with a corresponding set of one or more fitting parameters associated with a continuous functional model.

55. The video transcoding system of claim 49, wherein the side information stream includes N distortion information streams corresponding to N respective coding modes, wherein N is greater than or equal to one, wherein each distortion information stream DIS_k of the N distortion information streams characterizes a corresponding distortion estimation function D_k(q) per block of the first encoded video stream assuming block prediction based on the respective coding mode, wherein q is a quantization step size.

56. The video transcoding system of claim 55, wherein N=2 for a given coded picture of the first encoded video stream, wherein the distortion information stream DIS₁ corresponds to an intra coding mode, wherein the distortion information stream DIS₂ corresponds to an inter coding mode.

57. The video transcoding system of claim 55, wherein each distortion information stream DIS_k characterizes the distortion estimation function D_k(q) for each block with a corresponding set of one or more fitting parameters associated with a continuous functional model.

58. The video encoder of claim 49, wherein the output encoder is configured to process the side information stream in order to obtain an aggregate rate estimation function R_A(q) for each frame of the scaled video stream, wherein q represents quantization step size.

59. The video encoder of claim 58, wherein the output encoder is configured to further process the side information stream in order to obtain an aggregate distortion estimation function D_A(q) for each frame of the scaled video stream.

60. The video transcoding system of claim 49, wherein the side information stream includes one or more candidate motion vectors for each block of the first encoded video stream, wherein the output encoder is configured to perform a fine-resolution motion vector refinement for each block that is restricted to one or more neighborhoods in motion vector space based on the one or more candidate motion vectors.

61. The video transcoding system of claim 49, wherein said decoder is configured to recover a motion vector for each block from the first encoded video stream as part of said decoding the first encoded video stream, wherein the output encoder is configured to perform a motion vector refinement for each block that is restricted to a neighborhood in motion vector space based on the motion vector for the block.

62. The video transcoding system of claim 49, wherein the side information includes one or more candidate motion vectors per block, wherein said encoding the scaled video stream includes selecting a motion vector from a set of vectors including the one or more candidate motion vectors.

63. The video transcoding system of claim 62, wherein the set of vectors also include a decoded motion vector recovered from the first encoded video stream.

64. The video transcoding system of claim 49, further comprising:

transmission circuitry configured to transmit the second encoded video stream to a remote decoder through a communication medium.

65. The video transcoding system of claim 49, wherein the output encoder is configured to receive auxiliary information and inject the auxiliary information into the scaled video stream, wherein the auxiliary information includes one or more of:

branding information of a business entity;

advertising information;

digital rights management (DRM) information;

digital information providing watermark functionality;

customized features requested by a content provider, content delivery service provider, customer or user.

66. The video transcoding system of claim 49, wherein at least one of the decoder, the scaling unit and the output encoder is implemented using software configured for execution on an array of parallel processors.

67. The video transcoding system of claim 49, wherein the decoder, the scaling unit, the output encoder are implemented on distinct subsets of processors in an array of parallel processors.