Conditional concealment of lost video data

• US 9,609,321 B2
• Filed: 03/14/2013
• Issued: 03/28/2017
• Est. Priority Date: 01/28/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhao, David Yuheng, Nikiforov, Sergei, Hofbauer, Konrad, Hemmendorff, Magnus, Carlsson, Pontus
• Primary Examiner(s): Rahaman, Mohammed
• Assistant Examiner(s): Lee, Jimmy S
• Application Number:
  US13/803,738
• Publication Number:
  US 20140211860A1
• Time in Patent Office: 1,475 Days
• Field of Search: 375/240, 375/240.01, 375/240.26, 375/240.27
• US Class Current: 1/1
• CPC Class Codes: H04N 19/127 Prioritisation of hardware or computational resources
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/895 in combination with error concealment

Abstract

A receiving apparatus comprises a receiver for receiving a video signal, an output for outputting the video signal to a screen, storage device or further terminal; and a concealment module for applying a concealment algorithm to a lost area of the video signal. The concealment module is configured to determine an estimate of concealment quality selectively directed toward a region of interest within the area in question, and based on this estimate to determine whether or not to apply the concealment algorithm.

Claims

1. A receiving apparatus comprising:

a hardware receiver for receiving a video signal;

an output device for outputting the video signal to one or more of a screen, storage device or further terminal; and

a concealment module stored on one or more computer-readable storage memories and configured to be executable to apply a concealment algorithm to an area of said video signal having lost data, the concealment module being configured to determine an estimate of concealment quality selectively directed toward a region of interest within said area, the estimate based at least in part on a measure of difference between corresponding pixels in different frames of the video signal that surround a block at least on three sides, the block representing the region of interest in each of the different frames, the estimate excluding pixels on a side of the block different than the at least three sides responsive to determining that the excluded pixels are in another block having non-linear motion beyond a threshold of linearity, the concealment module configured to determine whether or not to apply the concealment algorithm based on said estimate.

2. The apparatus of claim 1, wherein the concealment module is configured, based on said estimate directed to the region of interest, to make said determination about whether or not to apply the concealment algorithm for the whole of said area, so as when applied to conceal data lost from the region of interest and data lost from a further region outside the region of interest.

3. The apparatus of claim 2, wherein the video signal comprises a plurality of frames, and said area comprises a whole frame, the concealment module making said determination about whether or not to apply the concealment algorithm once for the whole frame based on said estimate of concealment quality for the region of interest within that frame.

4. The apparatus of claim 1, wherein said region of interest comprises an image of at least part of a face.

5. The apparatus of claim 1, wherein the video signal comprises a live video stream.

6. The apparatus of claim 5, wherein the video stream is received as part of a live video call.

7. The apparatus of claim 1, wherein the concealment module is configured to determine the region of interest by receiving an indication of the region of interest received from a terminal transmitting the video signal.

8. The apparatus of claim 7, wherein said indication comprises a region-of-interest map received from the transmitting terminal.

9. The apparatus of claim 1, wherein the concealment module is configured to determine the region of interest by applying an image recognition algorithm at the receiving apparatus.

10. Apparatus of claim 9, wherein said image recognition algorithm comprises a facial recognition algorithm.

11. The apparatus of claim 2, wherein the concealment module is configured to determine said estimate of concealment quality only based on the region of interest, and to determine whether or not to apply the concealment algorithm over said area without taking into account concealment quality for the further region.

12. The apparatus of claim 2, wherein the concealment module is configured to determine said estimate of concealment quality based on the region of interest and the further region, but weighted towards the region of interest.

13. The apparatus of claim 1, wherein the video signal comprises a plurality of frames, and the concealment algorithm is configured to conceal the area by projecting replacement data from a past received frame.

14. The apparatus of claim 1, wherein the video signal comprises a plurality of frames, and the concealment algorithm is configured to conceal the area by projecting replacement data from a received part of a same frame or interpolating between received parts of the same frame.

15. The apparatus of claim 1, wherein the video signal comprises a plurality of frames, and the concealment algorithm is configured so as, when the concealment algorithm is not applied, to instead freeze at least a part of a previous frame for a duration.

16. The apparatus of claim 1, wherein the video signal comprises a plurality of frames received in encoded form, and the apparatus comprises a decoder for decoding the received frames of the video signal for output to the screen.

17. The apparatus of claim 1, wherein the receiver is configured to receive the video signal from a transmitter over a packet-based network.

18. A system comprising:

one or more processors; and

one or more computer-readable storage memories comprising computer-readable instructions for processing a video signal comprising a plurality of frames, which, when executed by the one or more processors, implement a method comprising;

processing the video signal for output to a screen, storage device or further terminal; and

conditionally applying a concealment algorithm over an area of said video signal having lost data, the conditional application of the concealment algorithm comprising;

determining an estimate of concealment quality selectively directed towards a region of interest within said area, the estimate based at least in part on a measure of difference between corresponding pixels in different frames of the video signal that surround a block at least on three sides, the block representing the region of interest in each of the different frames, the estimate excluding pixels on a side of the block different than the at least three sides responsive to determining that the excluded pixels are in another block having non-linear motion beyond a threshold of linearity; and

determining whether or not to apply the concealment algorithm based on said estimate.

19. One or more computer-readable storage memories comprising computer-readable instructions which, when executed on a terminal of a first user implement a method comprising:

establishing a live video call with a terminal of a second user over a packet-based network, and as part of the call receiving, in encoded form, a live video stream comprising a plurality of frames representing the second user;

decoding the received frames of the video stream and outputting the video stream to a screen; and

conditionally applying a concealment algorithm to a target frame of said video stream having lost data, the concealment algorithm projecting a replacement for the lost data from a previous received, decoded frame, the conditional application of the concealment algorithm comprising determining within the frame a region of interest comprising an image of at least part of the second user'"'"'s face, determining an estimate of concealment quality selectively directed towards said region of interest, the estimate based at least in part on a measure of difference between corresponding pixels in different frames comprising the video signal that surround a block at least on three sides, the block comprising the region of interest in each of the different frames, the estimate excluding pixels on a side of the block different than the at least three sides responsive to determining that the excluded pixels are in another block having non-linear motion beyond a threshold of linearity, and, based on said estimate, making a determination for the whole target frame as to whether or not to apply the concealment algorithm, so as when applied to conceal lost data from the region of interest and lost data from a region of the target frame outside the region of interest.

20. The one or more computer-readable storage memories of claim 19, wherein at least some of the pixels that surround the block are more than one pixel deep from an edge of the block.