Method for detecting and selecting good quality image frames from video

• US 7,916,173 B2
• Filed: 06/20/2005
• Issued: 03/29/2011
• Est. Priority Date: 06/22/2004
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ilbery, Peter William Mitchell, Politis, George, Clark, James David
• Primary Examiner(s): KOSTAK, VICTOR R
• Application Number:
  US11/155,632
• Publication Number:
  US 20050286802A1
• Time in Patent Office: 2,108 Days
• Field of Search: 348/700, 348/701, 348/180, 348/192, 348/189, 348/181, 348/722, 375/240.27, 375/240, 375224-228, 382/255, 382/232, 382/286, 358/527, 702/69, 702/71, 702/81
• US Class Current: 348/180
• CPC Class Codes: H04N 17/00 Diagnosis, testing or measuring for television systems or their details
• CPC Class Codes: H04N 17/06 for recorders
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Abstract

A method of determining a quality value for an image frame is disclosed. The method comprises dividing (in a step 202) the frame into a plurality of tiles (906) and determining attributes (in a step 206) of each said tile based upon pixel values of the tile, and pixel values of a corresponding tile of a preceding frame. The method then establishes (in steps 210, 212) the quality value of the frame by testing the tile attributes of the frame against pre-determined criteria. The method then defines (in a step 220) the quality value of the frame depending upon results of the testing.

Claims

1. A method of determining a quality value of a frame, the method comprising the steps of:

dividing the frame into a plurality of tiles;

determining a motion magnitude for each said tile based upon (i) pixel values of the tile and (ii) pixel values of a corresponding tile of a preceding frame;

determining a focus magnitude attribute for each tile of the plurality of tiles having a motion magnitude attribute below a predetermined threshold, the focus magnitude attribute being based on pixel values of the tile; and

establishing the quality value of the frame by;

(1) testing the motion and focus magnitude attributes of the frame against predetermined criteria; and

(2) defining the quality value of the frame depending upon results of said testing.

2. A method according to claim 1, wherein if the tiles of the frame do not meet the criteria of the testing step, then the defining step defines a default quality value for the frame.

3. A method according to claim 1, wherein the pixel value comprises the luminance value of a pixel which is represented in one of YUV and YCbCr video color space.

4. A method according to claim 1, wherein:

the frame is one of a sequence of video frames;

the method is applied to each frame in the sequence; and

the best frame in the sequence is identified as that frame having the best quality value.

5. A method according to claim 3, wherein the attributes comprise a motion definition state and a motion magnitude, these being determined by:

(1) classifying the tile as having a motion defined state if one of (i) an intensity range of pixels in the tile or in the corresponding tile of the preceding frame exceeds a pre-determined luminance threshold; and

(ii) a difference between mid-range pixel values for the tile and the corresponding tile of the preceding frame exceeds a pre-determined mid-range difference threshold; and

(2) determining, if the tile has a motion defined state, the motion magnitude dependent upon a difference in pixel value distributions between a pixel value histogram of the tile and a pixel value histogram of the corresponding tile in the previous frame, and

wherein the attributes further comprise a focus definition state and a focus magnitude, these being determined by;

(1) classifying the tile as having a focus undefined state if (i) the intensity range of pixels in the tile is too small, or there is too much motion in the tile or (ii) the number of defined difference quotients in the tile is less than a threshold; and

(2) determining the focus magnitude depending upon the largest difference quotient values of pixel luminance of the tile.

6. A method according to claim 5, wherein the pre-determined criteria are:

(a) that the number of tiles in the frame having a motion defined state exceed a pre-defined motion defined threshold;

(b) that the ratio of (i) tiles in the frame having a motion defined state whose motion magnitude exceeds a pre-defined motion threshold, to (ii) the total number of tiles in the frame having a motion defined state, exceeds a pre-defined motion defined threshold;

(c) that the number of tiles in the frame having a focus defined state exceeds a pre-defined focus defined threshold; and

(d) that the average of the largest focus magnitude values of tiles in the frame having a focus defined state exceeds a pre-defined focus defined threshold.

7. A method according to claim 6, wherein if the tiles of the frame meet all the criteria of the testing step, then the quality value depends upon the ratio and the average determined in the steps (b) and (d) of claim 6.

8. A method according to claim 1, wherein:

the quality value is determined based upon a sub-region of the frame; and

the dividing step divides the sub-region into the plurality of tiles.

9. A method according to claim 8, wherein the defining step is dependent upon a weight establishing the importance of the sub-region in the frame, and a weight establishing the size of the sub-region relative to the frame.

10. A method according to claim 6, wherein the criterion (b) is dependent upon the spatial location of tiles in the frame, and

wherein the criterion (b) is that the ratio of (i) tiles in a pre-defined region of the frame having a motion defined state whose motion magnitude exceeds a pre-defined motion threshold, to (ii) the total number of tiles in the frame having a motion defined state, exceeds a pre-defined motion defined threshold.

11. A method according to claim 6, wherein the criterion (d) is dependent upon the spatial location of tiles in the frame, and

wherein the criterion (d) is that the average of the largest focus magnitude values of tiles in the frame having a focus defined state exceeds a pre-defined focus defined threshold, wherein the focus magnitude values are weighted according to the relative distance of respective tiles from the center of the frame.

12. A method according to claim 7, wherein the quality value further depends upon a spatial distribution of the tiles having one of a motion defined state and a focus defined state.

13. An apparatus for determining a quality value of a frame, the apparatus comprising:

means for dividing the frame into a plurality of tiles;

means for determining a motion magnitude attribute for each said tile based upon (i) pixel values of the tile and (ii) pixel values of a corresponding tile of a preceding frame;

means for determining a focus magnitude attribute for each tile of the plurality of tiles having a motion magnitude attribute below a predetermined threshold, the focus magnitude attribute being based on pixel values of the tile; and

means for establishing the quality value of the frame comprising;

(1) means for testing the motion and focus magnitude attributes of the frame against predetermined criteria; and

(2) means for defining the quality value of the frame depending upon results of the testing.

14. A computer readable storage medium having a computer program recorded therein for directing a processor to execute a method of determining a quality value of a frame, said computer program comprising:

code for dividing the frame into a plurality of tiles;

code for determining a motion magnitude attribute for each said tile based upon (i) pixel values of the tile and (ii) pixel values of a corresponding tile of a preceding frame;

code for determining a focus magnitude attribute for each tile of the plurality of tiles having a motion magnitude attribute below a predetermined threshold, the focus magnitude attribute being based on pixel values of the tile; and

code for establishing the quality value of the frame comprising;

(1) code for testing the motion and focus attributes of the frame against predetermined criteria; and

(2) code for defining the quality value of the frame depending upon results of the testing.

15. A video frame selected from a sequence of video frames dependent upon a quality value of the frame, the frame being selected by a method of determining a quality value of a video frame, said method being applied to each frame in the sequence, said method comprising, in regard to a particular frame in the sequence, the steps of:

dividing the frame into a plurality of tiles;

determining a motion magnitude attribute for each said tile based upon (i) pixel values of the tile and (ii) pixel values of a corresponding tile of a preceding frame;

determining a focus magnitude attribute for each tile of the plurality of tiles having a motion magnitude attribute below a predetermined threshold, the focus magnitude attribute being based on pixel values of the tile; and

establishing the quality value of the frame by;

(1) testing the motion and focus attributes of the frame against predetermined criteria; and

(2) defining the quality value of the frame depending upon results of said testing.

34. A method according to claim 15, wherein said selecting at least one image is determined by ranking quality metric values and selecting the highest value image as the best image.

16. A method of estimating quality of an image in a video, the video comprising a plurality of images, each image comprising a plurality of pixels, said method comprising the steps of:

arranging, for a current image of the video, pixels of the current image into a plurality of tiles, each said tile containing a plurality of pixels;

determining the number of the tiles of the current image having defined characteristics; and

estimating quality of the current image based on the number of the tiles having the characteristics as a proportion of the number of tiles in the current image,

wherein the characteristics are (a) a motion characteristic, determined for each tile, based upon (i) pixel values of the tile and (ii) pixel values of a corresponding tile, of a preceding image, and (b) a focus characteristic, determined for each tile having a motion characteristic below a predetermined threshold, the focus characteristic being based on pixel values of the tile, and

wherein the motion characteristic establishes a motion magnitude in the tile, and the focus characteristic establishes a focus magnitude in the tile.

18. A method according to claim 16 or claim 17, wherein the characteristic is dependent on luminance.

19. A method according to claim 16 or claim 17, wherein the tiles of the current image comprise a predetermined number of pixels NP.

20. A method according to claim 16 or claim 17, wherein the characteristic is an exposure characteristic.

21. A method according to claim 20, wherein the motion characteristic classifies a tile as either motion-defined or motion-undefined.

22. A method according to claim 20, wherein the motion characteristic establishes the motion magnitude in the tile if the tile is determined to be motion-defined.

23. A method according to claim 20, wherein the focus characteristic classifies a tile as either focus-defined or focus-undefined.

24. A method according to claim 20, wherein the focus characteristic establishes the focus magnitude in the tile if the tile is determined to be focus-defined.

25. A method according to claim 21, wherein determining the motion characteristics of a tile comprises the steps of:

obtaining, for each color channel of tile pixels, a respective range of sample values; and

setting a motion characteristic of the tile of the current image to indicate that the tile is motion-undefined if;

(1) the color channel range of the tile of the current image does not exceed a color channel range threshold value, and (2) the color channel range of a corresponding tile of a previous image does not exceed the color channel range threshold value, and (3) the absolute difference between a color channel mid-range value of the tile of the current image and a corresponding color channel mid-range value of the corresponding tile of the previous image does not exceed a color channel mid-range difference threshold value; and

, otherwise, setting the motion characteristic of the tile of the current image to indicate that the tile is motion-defined.

26. A method according to claim 24, wherein determination of the focus characteristic of a tile comprises the steps of:

generating, for each sample of said color channel, a difference quotient for the sample value in the horizontal and vertical directions; and

defining the focus magnitude of the tile to be equal to the maximum absolute value of the difference quotient values of the tile if the tile of the current image is focus-defined.

27. A method according to claim 16, wherein determining the proportion comprises the steps of:

determining the respective numbers of tiles in the image having respective good, bad, and unknown classifications;

determining an image metric value for the image; and

classifying the frame as being one of acceptable and unacceptable dependent upon (i) the respective numbers of tiles in the image having good, bad, and unknown classification and (ii) the image metric value.

28. A method according to claim 27, wherein the image metric value is an image motion metric value defined as the number of motion-defined tiles with a motion magnitude greater than a motion magnitude threshold value, normalized against the total number of motion-defined tiles.

29. A method according to claim 27, wherein the image metric value is an image focus metric value defined as the average of the focus magnitudes of focus-defined tiles with a focus magnitude greater than a focus magnitude threshold value.

30. A method according to claim 27, wherein the image metric value is an image exposure metric value dependent upon the entropy of the luminance values of the frame.

31. A method according to claim 16, wherein the quality of the current image is defined as the weighted contribution from motion, focus, and exposure characteristic values.

32. A method according to claim 31, wherein the quality of a current image is assigned a normalized value of 0.0 for the worst quality image, 1.0 for the best quality image, and corresponding intermediate values for images of intermediate quality.

33. A method according to claim 32, wherein the quality of a current image is classified as acceptable if any of the following criteria are satisfied:

the proportion of known tiles exceeds a first threshold value;

the proportion of good tiles exceeds a second threshold value; and

the exposure metric exceeds a third threshold value.

17. A method of selecting an image from a video comprising a plurality of images, each image comprising a plurality of pixels, wherein said method comprises the steps of:

(a) performing, for each image of the plurality of images, the steps of;

(1) arranging pixels of the image into a plurality of tiles, each said tile containing a plurality of pixels;

(2) determining the number of the tiles of the image having defined characteristics; and

(3) estimating quality of the image based on the number of the tiles having the characteristics as a proportion of the number of tiles in the image; and

(b) selecting an image from the video in accordance with the estimated quality,

wherein the characteristics are (a) a motion characteristic, determined for each tile, based upon (i) pixel values of the tile and (ii) pixel values of a corresponding tile of a preceding image, and (b) a focus characteristic, determined for each tile having a motion characteristic below a predetermined threshold, the focus characteristic being based on pixel values of the tile, and

wherein the motion characteristic establishes a motion magnitude in the tile, and the focus characteristic establishes a focus magnitude in the tile.

35. A method of determining a quality value of a frame, the method comprising the steps of:

dividing the frame into a plurality of tiles;

determining attributes of each said tile based upon (i) pixel values of the tile and (ii) pixel values of a corresponding tile of a preceding frame, wherein the attributes comprise a motion definition state and a motion magnitude, the motion definition state being determined by classifying the tile as having a motion defined state if one of (i) an intensity range of pixels in the tile or in the corresponding tile of the preceding frame exceeds a pre-determined luminance threshold and (ii) a difference between mid-range pixel values for the tile and the corresponding tile of the preceding frame exceeds a pre-determined mid-range difference threshold, and determining the motion magnitude, if the tile has a motion defined state, based upon a difference in pixel value distributions between a pixel value histogram of the tile and a pixel value histogram of the corresponding tile in the previous frame; and

establishing the quality value of the frame by;

(1) testing the tile attributes of the frame against pre-determined criteria; and

(2) defining the quality value of the frame depending upon results of said testing, wherein;

the frame is one of a sequence of video frames;

the method is applied to each frame in the sequence; and

the best frame in the sequence is identified as that frame having the best quality value.

36. A method of determining a quality value of a frame, the method comprising the steps of:

dividing the frame into a plurality of tiles;

determining attributes of each said tile based upon pixel values of the tile, wherein the attributes comprise a focus definition state and a focus magnitude, the focus definition state being determined by classifying the tile as having a focus undefined state if (i) an intensity range of pixels in the tile is less than a predetermined threshold, or (ii) a number of defined difference quotients in the tile is less than a threshold, and determining the focus magnitude based on the largest difference quotient values of pixel luminance of the tile; and

establishing the quality value of the frame by;

(1) testing the tile attributes of the frame against pre-determined criteria; and

(2) defining the quality value of the frame depending upon results of said testing, wherein;

the frame is one of a sequence of video frames;

the method is applied to each frame in the sequence; and

the best frame in the sequence is identified as that frame having the best quality value.