System and method for automatic encoder adjustment based on transport data

• US 10,165,286 B2
• Filed: 07/31/2015
• Issued: 12/25/2018
• Est. Priority Date: 07/08/2009
• Status: Active Grant

• Current Assignee: Dejero Labs, Inc.
• Original Assignee: Dejero Labs, Inc.
• Inventors: Sze, David Pui Keung, Horvath, Akos, Frusina, Bogdan, Gilhuly, Barry, Smith, Cameron Kenneth, Mallet, Joseph Robert Wayne, Schneider, Anthony Todd, Flatt, Robert, Kaye, Hagen
• Primary Examiner(s): Duong, Oanh
• Application Number:
  US14/815,363
• Publication Number:
  US 20150341646A1
• Time in Patent Office: 1,243 Days
• CPC Class Codes: H04L 5/003 Arrangements for allocating sub-channels of the transmission path
• CPC Class Codes: H04L 5/0053 Allocation of signaling, i.e. of overhead other than pilot signals
• CPC Class Codes: H04L 5/0058 Allocation criteria
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/6131 involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/631 Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• CPC Class Codes: H04N 21/6373 for rate control , e.g. request to the server to modify its transmission rate flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/6377 directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 Transmission by the client directed to the server

Abstract

A system and method for transmission of a video stream are provided. The system may include: an encoder adapted to generate a video stream comprising a plurality of encoded frames, encoded according to at least one encoding parameter; a comparator in communication with the encoder, the comparator adapted to compare encoded frames of the plurality of encoded frames with input frames to determine a fitness metric reflective of visual quality of the encoded frames; and a controller in communication with the comparator, the controller adapted to adjust the at least one encoding parameter based on the fitness metric.

Claims

1. A system for transmission of a video stream across a plurality of connected networks, each network having a corresponding available bandwidth, the system comprising:

a connection controller circuit controlling a plurality of interfaces for transmitting the video stream over the plurality of connected networks, each interface corresponding to a specific network of the plurality of connected networks;

an encoder circuit adapted to generate individual video streams comprising a plurality of encoded frames, encoded according to at least one encoding parameter, each individual video stream transmitted to a corresponding interface of the plurality of interfaces for transmission across a corresponding network at an estimated maximum bit rate derived based on a predicted available bandwidth available across all of the plurality of connected networks;

a comparator circuit in communication with the encoder circuit, the comparator adapted to compare encoded frames of the plurality of encoded frames with input frames to determine a fitness metric reflective of visual quality of the encoded frames upon detecting a change in the predicted available bandwidth determined from feedback data obtained by the connection controller circuit corresponding to each network; and

an encoding controller circuit in cooperation with the comparator, the encoding controller circuit configured to;

receive data corresponding to the available bandwidth for each network of the plurality of connected networks, and

control the encoder circuit to adjust the at least one encoding parameter for each of the individual video streams based on the available bandwidth of the corresponding network and the fitness metric, the encoder circuit and the comparator cooperating to iteratively adjust the at least one encoding parameter until the fitness metric meets or exceeds a pre-defined minimum value to optimize visual quality of the video stream for a bit rate determined based at least on the available bandwidth of the corresponding network of the plurality of networks.

2. The system of claim 1, wherein the input frames are provided to the comparator circuit from a video capture device.

3. The system of claim 1, further comprising a pre-processor that processes frames prior to encoding, and wherein the input frames are provided to the comparator circuit from the pre-processor.

4. The system of claim 3, wherein the input frames are scaled versions of frames captured by a video capture device.

5. The system of claim 1, further comprising a transmitter adapted to transmit encoded frames over a network.

6. The system of claim 1, wherein the comparator circuit applies at least one of Structural Similarity (SSIM), Peak Signal to Noise Ratio (PSNR), Mean Squared Error (MSE), and Multi-Scale SSIM (MS-SSIM) to determine the fitness metric.

7. The system of claim 1, wherein the encoding controller circuit is adapted to adjust the at least one encoding parameter based on a complexity level of the video stream.

8. The system of claim 7, wherein the complexity level of the video stream is determined based on motion estimation analysis of the input frames of the video stream.

9. The system of claim 7, wherein the complexity level is based on an indicator identifying a type of broadcast being encoded to generate the video stream.

10. The system of claim 7, wherein when the video stream has a high complexity level, adjusting the at least one encoding parameter comprises:

increasing a frame rate, and decreasing a frame size to accommodate the available bandwidth.

11. The system of claim 1, wherein the encoding controller circuit is configured for adjusting the fitness metric based on a complexity level of the video stream.

12. The system of claim 1, wherein the input frames compared by the comparator circuit to determine the fitness metric are pre-processed and scaled down input frames.

13. The system of claim 1, wherein, the encoded frames compared by the comparator circuit to determine the fitness metric are encoded frames re-scaled to an original size of the input frames.

14. A method for transmitting a video stream across a plurality of connected networks, each network having a corresponding available bandwidth, the method comprising:

controlling a plurality of interfaces for transmitting the video stream over the plurality of connected networks, each interface corresponding to a specific network of the plurality of connected networks;

generating individual video streams each comprising a plurality of the encoded frames;

encoding the frames of the video stream according to at least one encoding parameter for each of network of the plurality of connected networks for transmission across a corresponding network at an estimated maximum bit rate derived based on a predicted available bandwidth available across all of the plurality of connected networks;

comparing the encoded frame with an input frame to determine a fitness metric reflective of visual quality of the encoded frame upon detecting a change in the predicted available bandwidth determined from feedback data obtained by the connection controller circuit corresponding to each network;

receiving data corresponding to the available bandwidth for each network of the plurality of connected networks over which the video stream is transmitted; and

controlling an encoder to adjust the at least one encoding parameter for each of the individual video streams based on the available bandwidth of the corresponding network and the fitness metric, the encoder cooperating with a comparator to iteratively adjust the at least one encoding parameter until the fitness metric meets or exceeds a pre-defined minimum value to optimize visual quality of the video stream for a bit rate determined based at least on the available bandwidth of the corresponding network of the plurality of networks.

15. The method of claim 14, further comprising:

encoding a further frame of the video stream according to the adjusted at least one encoding parameter.

16. The method of claim 14, further comprising:

pre-processing the input frame, the pre-processing comprising scaling the input frame.

17. The method of claim 14, wherein the fitness metric is determined based on at least one of Structural Similarity (SSIM), Peak Signal to Noise Ratio (PSNR), Mean Squared Error (MSE), and Multi-Scale SSIM (MS-SSIM).

18. The method of claim 14, further comprising adjusting the at least one encoding parameter based on a correlated parameter.

19. The method of claim 18, wherein the correlated parameter is a packet loss rate.

20. The method of claim 14, further comprising adjusting the at least one encoding parameter based on a maximum bit rate.

21. The method of claim 14, further comprising adjusting the at least one encoding parameter based on a complexity level of the video stream.

22. The method of claim 21, wherein the complexity level of the video stream is determined based on motion estimation analysis of the input frames of the video stream.