Keyframe mitigation for video streams with multiple receivers

• US 10,575,035 B2
• Filed: 12/04/2018
• Issued: 02/25/2020
• Est. Priority Date: 11/21/2016
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Bright-Thomas, Paul G.
• Primary Examiner(s): Dubasky, Gigi L
• Application Number:
  US16/208,689
• Publication Number:
  US 20190104327A1
• Time in Patent Office: 448 Days
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/65 using error resilience
• CPC Class Codes: H04N 19/895 in combination with error concealment
• CPC Class Codes: H04N 21/234 Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
• CPC Class Codes: H04N 21/2365 Multiplexing of several video streams
• CPC Class Codes: H04N 21/2407 Monitoring of transmitted content, e.g. distribution time, number of downloads arrangements for monitoring programmes for broadcast or for distribution combined with broadcast H04H20/14
• CPC Class Codes: H04N 21/6379 directed to encoder , e.g. for requesting a lower encoding rate

Abstract

In one embodiment, a video encoder generates an encoded bitstream representing a sequence of video frames including a keyframe. The encoder generates information for use by a decoder that receives the encoded bitstream to enable the decoder to generate display frames from a pre-keyframe video frame that is prior to the keyframe in the sequence of video frames. The encoded bitstream is sent to the decoder. In another embodiment, a video decoder receives from an encoder an encoded bitstream representing a sequence of video frames including a keyframe. The keyframe includes information to enable the decoder to generate display frames from a pre-keyframe video frame that was received prior to the keyframe in the sequence of video frames. The decoder generates display frames using the pre-keyframe video frame, information included with the keyframe and information included with an encoder-determined number of decoded frames subsequent to the keyframe.

Claims

1. A method comprising:

at a video decoder, receiving from an encoder an encoded bitstream representing a sequence of video frames, the sequence of video frames including a keyframe and at least one pre-keyframe video frame received prior to the keyframe in the sequence of video frames, wherein the keyframe includes improvement data generated by the encoder to enable the video decoder to generate display frames based on the at least one pre-keyframe video frame that was received prior to the keyframe;

generating one or more improvement frames based on the at least one pre-keyframe video frame using the improvement data included with the keyframe, wherein the improvement data is related to motion compensation;

generating a predetermined number of display frames subsequent to the keyframe, wherein each of the predetermined number of display frames is generated using a combination of an improvement frame from the one or more improvement frames and a decoded frame; and

upon completion of generating the predetermined number of display frames, generating additional subsequent display frames using a decoded frame without an improvement frame.

2. The method of claim 1, wherein the improvement data comprises motion compensation relative to one or more reference frames prior to the keyframe.

3. The method of claim 2, wherein the motion compensation comprises a differential motion vector expression relative to a predictor motion vector.

4. The method of claim 2, wherein the motion compensation comprises one or more skip blocks having identical motion to one or more predictor motion vectors.

5. The method of claim 2, further comprising using residual data to generate the one or more improvement frames.

6. The method of claim 2, further comprising applying de-blocking or smoothing filter operations between blocks of the one or more reference frames with divergent motion.

7. The method of claim 1, further comprising blending at boundaries between regions of the decoded frame that have improvement data and regions of the decoded frame that do not have improvement data.

8. A method comprising:

at a video encoder, generating an encoded bitstream representing a sequence of video frames, the sequence of video frames including a keyframe and at least one pre-keyframe video frame that is prior to the keyframe in the sequence of video frames;

generating improvement data for use by a decoder that receives the encoded bitstream to enable the decoder to generate display frames based on the at least one pre-keyframe video frame that is prior to the keyframe in the sequence of video frames, wherein the improvement data enables the decoder to generate one or more improvement frames based on the at least one pre-keyframe video frame using the improvement data included with the keyframe, wherein the improvement data is related to motion compensation and enables the decoder to generate a predetermined number of display frames subsequent to the keyframe, and wherein each of the predetermined number of display frames is generated by the decoder using a combination of an improvement frame from the one or more improvement frames and a decoded frame; and

including the improvement data with the keyframe and with a video encoder-determined number of frames subsequent to the keyframe.

9. The method of claim 8, wherein the improvement data comprises motion compensation relative to one or more reference frames prior to the keyframe.

10. The method of claim 9, wherein the motion compensation comprises a differential motion vector expression relative to a predictor motion vector.

11. The method of claim 9, wherein the motion compensation comprises one or more skip blocks having identical motion to one or more predictor motion vectors.

12. The method of claim 9, further comprising including residual data in the encoded bitstream to enable the decoder to generate the one or more improvement frames.

13. The method of claim 8, wherein decoders that are capable of interpreting the improvement data and decoders that are not capable of interpreting the improvement data are configured to decode the keyframe and the video encoder-determined number of frames.

14. An apparatus comprising:

a communication interface configured to enable communications over a network; and

a processor coupled to the communication interface, and configured to;

receive, via the communication interface, from an encoder an encoded bitstream representing a sequence of video frames, the sequence of video frames including a keyframe and at least one pre-keyframe video frame received prior to the keyframe in the sequence of video frames, wherein the keyframe includes improvement data generated by the encoder to enable the processor to generate display frames based on the at least one pre-keyframe video frame that was received prior to the keyframe;

generate one or more improvement frames based on the at least one pre-keyframe video frame using the improvement data included with the keyframe, wherein the improvement data is related to motion compensation;

generate a predetermined number of display frames subsequent to the keyframe, wherein each of the predetermined number of display frames is generated using a combination of an improvement frame from the one or more improvement frames and a decoded frame; and

upon completion of generating the predetermined number of display frames, generate additional subsequent display frames using a decoded frame without an improvement frame.

15. The apparatus of claim 14, wherein the improvement data comprises motion compensation relative to one or more reference frames prior to the keyframe.

16. The apparatus of claim 15, wherein the motion compensation comprises a differential motion vector expression relative to a predictor motion vector.

17. The apparatus of claim 15, wherein the motion compensation comprises one or more skip blocks having identical motion to one or more predictor motion vectors.

18. The apparatus of claim 15, wherein the processor is further configured to use residual data to generate the one or more improvement frames.

19. The apparatus of claim 15, wherein the processor is further configured to apply de-blocking or smoothing filter operations between blocks of the one or more reference frames with divergent motion.

20. The apparatus of claim 14, wherein the processor is further configured to blend at boundaries between regions of the decoded frame that have improvement data and regions of the decoded frame that do not have improvement data.