Adaptive region of interest

• US 9,262,670 B2
• Filed: 02/10/2012
• Issued: 02/16/2016
• Est. Priority Date: 02/10/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Pietil, Sami Aleksi
• Primary Examiner(s): Anyikire, Chikaodili E.
• Assistant Examiner(s): KALAPODAS, DRAMOS
• Application Number:
  US13/370,530
• Publication Number:
  US 20130208784A1
• Time in Patent Office: 1,467 Days
• Field of Search: 375/240.02, 375/240.03, 375/240.05, 382/168, 382232-253
• US Class Current: 1/1
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component

Abstract

This disclosure relates to adaptively determining and improving the quality of a region of interest in video content. A region inspection component inspects regions of an image. A detection component determines chroma values contained in the regions. A comparison component compares the chroma values against a set of predetermined chroma values, and determines, based on the comparison, a set of regions of interest in the frame. An encoder encodes the regions of interest in the image at a higher or better quality than a remainder of the image.

Claims

1. A method comprising:

generating, by a processor in response to instructions stored on a non-transitory computer readable medium, an encoded frame by encoding a current input frame from a plurality of frames of an input video stream, wherein encoding the current input frame includes;

identifying a current block from the current input frame,

determining a first chrominance value for the current block,

determining a second chrominance value for the current block,

determining whether the current block is a skin tone block, wherein the current block is a skin tone block on a condition that the first chrominance value intersects with the second chrominance value within a skin tone area in a chrominance map, and on a condition that an aggregate color value of the current block exceeds a filtering threshold, and on a condition that a cardinality of a plurality of neighboring skin tone blocks exceeds a spatial threshold, and wherein determining whether the current block is a skin tone block includes determining whether the first chrominance value intersects with the second chrominance value within the skin tone area in the chrominance map, wherein determining whether the first chrominance value intersects with the second chrominance value within the skin tone area in the chrominance map includes;

identifying a first chrominance range based on the first chrominance value;

identifying a second chrominance range based on the second chrominance value;

on a condition that the first chrominance value is within the second chrominance range and on a condition that the second chrominance value is within the first chrominance range, determining that the first chrominance value intersects with the second chrominance value within the skin tone area in the chrominance map;

on a condition that the first chrominance value is outside the second chrominance range, determining that the first chrominance value intersects with the second chrominance value outside the skin tone area in the chrominance map; and

on a condition that the second chrominance value is outside the first chrominance range, determining that the first chrominance value intersects with the second chrominance value outside the skin tone area in the chrominance map,

determining whether to include the current block in a region of interest for the current input frame based on whether the current block is a skin tone block and whether the current block spatially corresponds with a motion block in a motion map, wherein the motion map represents motion identified for at least one previously coded frame,

encoding the current block using region of interest information identifying the region of interest for the current input frame on a condition that a difference between the region of interest for the current input frame and a region of interest for the at least one previously coded frame exceeds an update threshold, wherein encoding the current block includes including information identifying the region of interest for the current input frame in an output bitstream,

encoding the current block using region of interest information identifying the region of interest for the at least one previously coded frame on a condition that the difference between the region of interest for the current input frame and the region of interested for the at least one previously coded frame is within the update threshold, and

including the encoded block in the output bitstream; and

transmitting or storing the output bitstream.

2. The method of claim 1, wherein encoding the current input frame includes identifying the skin tone chrominance map based on a plurality of training frames, wherein the input video sequence does not include the plurality of training frames.

3. The method of claim 1, wherein encoding the current input frame includes identifying the skin tone chrominance map based on a plurality of training frames, wherein the input video sequence does not include the plurality of training frames, and wherein a training frame from the plurality of training frames includes a training skin tone pixel such that the skin tone chrominance map does not include an intersection of chrominance values of the training skin tone pixel.

4. The method of claim 1, wherein the aggregate color value of the current block exceeds the filtering threshold on a condition that a color variance of the current block is greater than a minimum color variance.

5. The method of claim 1, wherein the aggregate color value of the current block exceeds the filtering threshold on a condition that a median absolute deviation of the current block is greater than a minimum median absolute deviation.

6. The method of claim 1, wherein encoding the current input frame includes identifying the motion map by:

updating a motion value of a block from the motion map on a condition that a motion vector for a spatially corresponding block from the at least one previously coded frame is a non-zero motion vector.

7. The method of claim 1, wherein the motion map is a cumulative motion map that indicates at least one spatial area of motion for the current input frame based on motion information generated for the at least one previously coded frame.

8. The method of claim 1, wherein determining whether to include the current block in the region of interest for the current input frame includes:

determining a region of interest score for the current block; and

including the current block in the region of interest for the current input frame on a condition that the region of interest score exceeds a region of interest threshold.

9. The method of claim 8, wherein determining the region of interest score includes:

determining a skin map block score;

determining a motion block score; and

determining a sum of the skin map block score and the motion block score as the region of interest score.

10. The method of claim 9, wherein the skin map block score is zero on a condition that the current block is not a skin tone block and the skin map block score is a skin map weight on a condition that the current block is a skin tone block.

11. The method of claim 9, wherein the motion block score is zero on a condition that the current block does not spatially correspond with a motion block from the motion map and the motion block score is a motion weight on a condition that the current block spatially corresponds with a motion block from the motion map.

12. The method of claim 1, wherein encoding the current input frame includes:

determining the difference between the region of interest for the current input frame and the region of interest for the at least one previously coded frame based a size of the region of interest for the current input frame, a size of region of interest for the at least one previously coded frame, a proportion of skin map blocks in the region of interest for the current input frame relative to the size of the region of interest for the current input frame, or a proportion of skin map blocks in the region of interest for the at least one previously coded frame relative to the size of the region of interest for the at least one previously coded frame.

13. A method comprising:

generating a first region of interest based on a first frame from a first input video stream;

generating a first motion map representing motion identified for the first frame;

generating a first encoded frame by encoding the first frame based on the first region of interest;

including information identifying the first region of interest in an output bitstream;

including the first encoded frame in the output bitstream;

identifying a second frame from the input video stream;

generating, by a processor in response to instructions stored on a non-transitory computer readable medium, a second encoded frame by encoding the second frame, wherein encoding the second frame includes;

identifying a current block from the second frame,

determining a first chrominance value for the current block,

determining a second chrominance value for the current block,

determining whether the current block is a skin tone block, wherein the current block is a skin tone block on a condition that the first chrominance value intersects with the second chrominance value within an identified skin tone area in a chrominance map, and on a condition that an aggregate color value of the current block exceeds a filtering threshold, and on a condition that a cardinality of a plurality of neighboring skin tone blocks from the second frame exceeds a spatial threshold, wherein determining whether the current block is a skin tone block includes determining whether the first chrominance value intersects with the second chrominance value within the identified skin tone area in the chrominance map, wherein determining whether the first chrominance value intersects with the second chrominance value within the identified skin tone area in the chrominance map includes;

identifying a first chrominance range based on the first chrominance value;

identifying a second chrominance range based on the second chrominance value;

on a condition that the first chrominance value is within the second chrominance range and on a condition that the second chrominance value is within the first chrominance range, determining that the first chrominance value intersects with the second chrominance value within the identified skin tone area in the chrominance map;

on a condition that the first chrominance value is outside the second chrominance range, determining that the first chrominance value intersects with the second chrominance value outside the identified skin tone area in the chrominance map; and

on a condition that the second chrominance value is outside the first chrominance range, determining that the first chrominance value intersects with the second chrominance value outside the identified skin tone area in the chrominance map,

generating a second motion map based on the first motion map and the second frame,

generating a second region of interest based on the second frame, wherein generating the second region of interest includes determining whether to include the current block in the second region of interest based on whether the current block is a skin tone block and whether the current block spatially corresponds with a motion block in the second motion map,

determining whether to encode the current block using the first region of interest or the second region of interest,

encoding the current block using the second region of interest on a condition that a difference between the second region of interest and the first region of interest exceeds an update threshold, wherein encoding the current block includes including information identifying the second region of interest in the output bitstream,

encoding the current block using the first region of interest on a condition that the difference between the second region of interest and the first region of interest is within the update threshold, and

including the second encoded block in the output bitstream; and

transmitting or storing the output bitstream.

14. The method of claim 13, wherein the aggregate color value of the current block exceeds the filtering threshold on a condition that a color variance of the current block is greater than a minimum color variance or on a condition that a median absolute deviation of the current block is greater than a minimum median absolute deviation.

15. The method of claim 13, wherein determining whether to include the current block in the second region of interest includes:

determining a skin map block score, wherein the skin map block score is zero on a condition that the current block is not a skin tone block and the skin map block score is a skin map weight on a condition that the current block is a skin tone block;

determining a motion block score, wherein the motion block score is zero on a condition that the current block does not spatially correspond with a motion block from the second motion map and the motion block score is a motion weight on a condition that the current block spatially corresponds with a motion block from the second motion map; and

including the current block in the second region of interest on a condition that a sum of the skin map block score and the motion block score exceeds a region of interest threshold.

16. The method of claim 13, wherein encoding the second frame includes determining the difference between the second region of interest and the first region of interest based a size of the second region of interest, a size of the first region of interest, a proportion of skin map blocks in the second region of interest relative to the size of the second region of interest, or a proportion of skin map blocks in the first region of interest relative to the size of the first region of interest.

17. A method comprising:

generating, by a processor in response to instructions stored on a non-transitory computer readable medium, an encoded frame by encoding a current frame from an input video stream, the current frame including a plurality of blocks, wherein encoding the current frame includes;

generating a motion map for the current frame based on a respective motion vector for each block from the plurality of blocks and motion information identified for at least one previously coded frame,

generating a tone map for the current frame, wherein generating the tone map includes;

identifying a subset of blocks from the plurality of blocks, wherein each block from the subset of blocks has chrominance values that intersect within a defined skin tone area in a chrominance map, wherein identifying the subset of blocks includes;

for each block from the subset of blocks, determining whether a first chrominance value for the block intersects with a second chrominance value for the block within the skin tone area in the chrominance map, wherein determining whether the first chrominance value intersects with the second chrominance value within the skin tone area in the chrominance map includes;

 

identifying a first chrominance range based on the first chrominance value;

 

identifying a second chrominance range based on the second chrominance value;

 

on a condition that the first chrominance value is within the second chrominance range and on a condition that the second chrominance value is within the first chrominance range, determining that the first chrominance value intersects with the second chrominance value within the skin tone area in the chrominance map;

 

on a condition that the first chrominance value is outside the second chrominance range, determining that the first chrominance value intersects with the second chrominance value outside the skin tone area in the chrominance map; and

 

on a condition that the second chrominance value is outside the first chrominance range, determining that the first chrominance value intersects with the second chrominance value outside the skin tone area in the chrominance map;

filtering the subset of blocks based on a color filtering threshold; and

identifying a spatial group of blocks from the subset of blocks, such that a cardinality of the spatial group of blocks exceeds a spatial threshold, such that the tone map identifies the spatial group of blocks;

identifying a region of interest for the current frame based on the motion map and the tone map,

including information identifying the region of interest for the current frame in an output bitstream on a condition that a difference between the region of interest for the current frame and a region of interest for a previously coded frame exceeds an update threshold, and

including the encoded block in the output bitstream; and

transmitting or storing the output bitstream.