Quality metric-biased region-of-interest coding for video telephony

• US 7,724,972 B2
• Filed: 08/09/2005
• Issued: 05/25/2010
• Est. Priority Date: 03/01/2005
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: El-Maleh, Khaled Helmi, Wang, Haohong
• Primary Examiner(s): Alavi; Amir
• Application Number:
  US11/199,773
• Publication Number:
  US 20060238444A1
• Time in Patent Office: 1,750 Days
• Field of Search: 382/173, 382/219, 382/224, 382/232, 382/233, 382/239, 382/243, 382/251, 382/254, 382/275, 375/240.04, 375/E7.081, 375/E7.088, 375/E7.128, 375/E7.14, 375/240.24
• US Class Current: 382/239
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/126 Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/198 including smoothing of a sequence of encoding parameters, e.g. by averaging, by choice of the maximum, minimum or median value
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

The disclosure is directed to techniques for region-of-interest (ROI) coding for video telephony (VT). The disclosed techniques include a technique for generation of a quality metric for ROI video, which jointly considers a user'"'"'s degree of interest in the ROI, ROI video fidelity, and ROI perceptual quality in evaluating the quality of an encoded video sequence. The quality metric may be used to bias ROI coding and, in particular, the allocation of coding bits between ROI and non-ROI areas of a video frame.

Claims

1. A method comprising generating, using a video encoding system, a quality metric for an encoded video frame containing a region of interest based on video fidelity of a preceding frame, perceptual quality of the preceding frame, and user preference for the region of interest, wherein the user preference indicates a value placed on visual quality of the region of interest by a user and further comprising allocating encoding bits between the region of interest and areas of the video frame that are not within the region of interest based at least in part on the quality metric.

2. The method of claim 1, wherein the perceptual quality includes a temporal quality value and a spatial quality value for the preceding frame.

3. The method of claim 2, wherein the temporal quality value includes a first temporal quality value for the region of interest, and a second temporal quality value for areas that are not within the region of interest.

4. The method of claim 2, wherein the spatial quality value includes a first spatial quality value for the region of interest, and a second spatial quality value for areas that are not within the region of interest.

5. The method of claim 2, wherein the spatial quality value is based at least in part on presence of blocking artifacts in the preceding frame, and wherein the temporal quality value is based at least in part on presence of temporal flicker artifacts in the preceding frame.

6. The method of claim 1, wherein the video fidelity includes a distortion value based at least in part on a peak signal to noise ratio for the preceding video frame.

7. The method of claim 1, wherein allocating encoding bits includes allocating rho domain values to macroblocks within the frame based on a frame budget and a weighting between macroblocks within the region of interest and macroblocks within areas of the video frame that are not within the region of interest, and mapping the rho domain values to corresponding quantization parameter (QP) values to allocate a number of encoding bits to each of the macroblocks.

8. The method of claim 1, wherein the quality metric comprises a rate budget specifying the number of encoding bits available for the video frame, and an indication of whether encoding of the areas that are not within the region of interest will be skipped.

9. The method of claim 1, further comprising grouping successive frames into a frame unit, encoding regions of interest associated with the respective frames, and skipping encoding of areas that are not within a respective reason of interest for at least one of the frames in the frame unit.

10. The method of claim 9, further comprising selectively skipping encoding of areas that are not within a respective region of interest for at least one of the frames when a distortion value associated with a preceding frame unit is less than a threshold value.

11. The method of claim 1, further comprising obtaining the user preference as a user preference value from a user, wherein the user preference value indicates a perceptual importance of visual quality of the region of interest from the perspective of the user.

12. The method of claim 1, wherein the encoded video frame is a first video frame, the method further comprising encoding a second video frame based on the allocation of encoding bits.

13. A device comprising:

a video encoder that encodes a video frame containing a region of interest; and

a quality metric calculator that generates a quality metric for the video frame based on video fidelity preceding frame, perceptual quality of a preceding frame, and user preference for the region of interest, wherein the user preference indicates a value placed on visual quality of the region of interest by a user and further comprising a bit allocation module that allocates encoding bits between the region of interest and areas that are not within the region of interest based at least in part on the quality metric.

14. The device of claim 13, wherein the perceptual quality includes a temporal quality value and a spatial quality value for the preceding frame.

15. The device of claim 14, wherein the temporal quality value includes a first temporal quality value for the region of interest, and a second temporal quality value for areas that are not within the region of interest.

16. The device of claim 14, wherein the spatial quality value includes a first spatial quality value for the region of interest, and a second spatial quality value for areas that are not within the region of interest.

17. The device of claim 14, wherein the spatial quality value is based at least in part on presence of blocking artifacts in the preceding frame, and wherein the temporal quality value is based at least in part on presence of temporal flicker artifacts in the preceding frame.

18. The device of claim 13, wherein the video fidelity includes a distortion value based at least in part on a peak signal to noise ratio for the preceding video frame.

19. The device of claim 13, wherein the bit allocation module allocates rho domain values to macroblocks within the frame based on a frame budget and a weighting between macroblocks within the region of interest and macroblocks within areas of the video frame that are not within the region of interest, and maps the rho domain values to corresponding quantization parameter (QP) values to allocate a number of encoding bits to each of the macroblocks.

20. The device of claim 13, wherein the quality metric comprises a rate budget specifying the number of encoding bits available for the video frame, and an indication of whether encoding of the areas that are not within the region of interest will be skipped.

21. The device of claim 13, further comprising a skipping module that directs the video encoder to group successive frames into a frame unit, encode regions of interest associated with the respective frames, and skip encoding of areas of the video frame that are not within a respective region of interest for at least one of the frames within the frame unit.

22. The device of claim 21, wherein the skipping module directs the video encoder to selectively skip encoding of areas of the video frame that are not within a respective region of interest for at least one of the frames when a distortion value associated with a preceding frame unit is less than a threshold value.

23. The device of claim 13, further comprising a wireless transmitter that transmits the encoded video frame via a wireless communication channel, wherein the device is configured to support mobile video telephony.

24. The device of claim 13, further comprising a user preference input device that obtains he user preference as a user preference value from a user, wherein the user preference value indicates a perceptual importance of visual quality of the region of interest from the perspective at the user.

25. The device of claim 13, wherein the device comprises an integrated circuit device.

26. The device of claim 13, wherein the encoded video frame is a first video frame, and the video encoder encodes a second video frame based on the allocation of encoding bits.

27. A computer-readable medium comprising instructions to cause a processor to generate a quality metric for an encoded video frame containing a region of interest based on video fidelity of a preceding frame, perceptual quality of the preceding frame, and user preference for the region of interest, wherein the user preference indicates a value placed on visual quality of the region of interest by a user and further comprising instructions to cause the processor to allocate encoding bits between the region of interest and areas of the video frame that are not within the region of interest based at least in part on the quality metric.

28. The computer-readable medium of claim 27, wherein the perceptual quality includes a temporal quality value and a spatial quality for the preceding frame.

29. The computer-readable medium of claim 28, wherein the temporal quality value includes a first temporal quality value for the region of interest, and a second temporal quality value for areas that are not within the region of interest.

30. The computer-readable medium of claim 28, wherein the spatial quality value includes a first spatial quality value for the region of interest, and a second spatial quality value for areas that are not within the region of interest.

31. The computer-readable medium of claim 28, wherein the spatial quality value is based at least in part on presence of blocking artifacts in the preceding frame, and wherein the temporal quality value is based at least in part on presence of temporal flicker artifacts in the preceding frame.

32. The computer-readable medium of claim 27, wherein the video fidelity includes a distortion value based at least in part on a peak signal to noise ratio for the preceding video frame.

33. The computer-readable medium of claim 27, wherein the instructions cause the processor to allocate rho domain values to macroblocks within the frame based on a frame budget and a weighting between macroblocks within the region of interest and macroblocks within areas of the video frame that are not within the region of interest, and map the rho domain values to corresponding quantization parameter (QP) values to allocate a number of encoding bits to each of the macroblocks.

34. The computer-readable medium of claim 27, wherein the quality metric comprises a rate budget specifying the number of encoding bits available for the video frame, and an indication of whether encoding of the areas that are not within the region of interest will be skipped.

35. The computer-readable medium of claim 27, further comprising instructions to cause the processor to group successive frames into frame unit, encode regions of interest associated with the respective frames, and skip encoding of areas that are not within a respective region of interest for at least one of the frames in the frame unit.

36. The computer-readable medium of claim 35, further comprising instructions to cause the processor to selectively skip encoding of areas that are not within a respective region of interest for at least one of the frames when a distortion value associated with a preceding frame unit is less than a threshold value.

37. The computer-readable medium of claim 27, further comprising instructions to cause the processor to obtain the user preference as a user preference value from a user, wherein the user preference value indicates a perceptual importance of visual quality of the region of interest from the perspective of the user.

38. The computer-readable medium of claim 27, wherein the encoded video frame is a first video frame, the computer-readable medium further comprising instructions to cause the processor to encode a second video frame based on the allocation of encoding bits.

39. A device comprising:

means for defining a region of interest for an encoded video frame; and

means for generating a quality metric for the encoded video frame containing the region of interest based on video fidelity of a preceding frame, perceptual quality of the preceding frame, and user preference for the region of interest, wherein means for obtaining the user preference as a user preference value from a user, wherein the user preference value indicates a perceptual importance of visual quality of the region of interest from the perspective of the user and further comprising means for allocating encoding bits between the region of interest and areas of the video frame that are not within the region of interest based at least in part on the quality metric.

40. The device of claim 39, wherein the perceptual quality includes a temporal quality value and a spatial quality value for the preceding frame.

41. The device of claim 40, wherein the temporal quality value includes a first temporal quality value for the region of interest, and a second temporal quality value for areas that are not within the region of interest.

42. The device of claim 40, wherein the spatial quality value includes a first spatial quality value for the region of interest, and a second spatial quality value for areas that are not within the region of interest.

43. The device of claim 40, wherein the spatial quality value is based at least in part on presence of blocking artifacts in the preceding frame, and wherein the temporal quality value is based at least in part on presence of temporal flicker artifacts in the preceding frame.

44. The device of claim 39, wherein the video fidelity includes a distortion value based at least in part on a peak signal to noise ratio for the preceding video frame.

45. The device of claim 39, wherein the encoded video frame is a first video frame, the device further comprising means for encoding a second video frame based on the allocated bits.

46. The device of claim 39, further comprising means for allocating rho domain values to macroblocks within the frame based on a frame budget and a weighting between macroblocks within the region of interest and macroblocks within areas of the video frame that are not within the region of interest, and mapping the rho domain values to corresponding quantization parameter (QP) values to allocate a number of encoding bits to each of the macroblocks.

47. The device of claim 39, wherein the quality metric comprises a rate budget specifying the number of encoding bits available for the video frame, and an indication of whether encoding of the areas that are not within the region of interest will be skipped.

48. The device of claim 39, further comprising means for grouping successive frames into frame unit, encoding regions of interest associated with the respective frames, and skipping encoding of areas that are not within a respective region of interest for at least one of the frames in the frame unit.

49. The device of claim 39, further comprising means for selectivity skipping encoding of areas that are not within a respective region of interest for at least one of the frames when a distortion value associated wilt a preceding frame unit is less than a threshold value.