Visual language classification system

• US 20030002715A1
• Filed: 12/07/2000
• Published: 01/02/2003
• Est. Priority Date: 12/14/1999
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kowald, Julie Rae
• Granted Patent: US 7,606,397 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

Disclosed is a method and system (500) for automated classification of a digital image (502). The method analyses the image for the presence of a human face. A determination is then made regarding the size of the located face compared to the size of the image(FIGS. 1A-1G) to classify the image based on the relative size of the face. Alternatively, the position of the face within the image can be used to determine the classification. With a classified image, particularly forming part of a sequence of classified images, editing (514) of the sequence may be performed dependent upon the classification to achieve a desired aesthetic effect. The editing may be performed with the aid of an editing template (706).

Claims

1. A method for automated classification of a digital image, said method comprising the steps of:

analysing said image for the presence of a human face;

determining a size of the located face with respect to a size of said image; and

classifying said image based on the relative size of said face with respect to said image.

2. A method according to claim 1 wherein said image is classified using a term which provides information about an intention of a photographer whom captured said image.

3. A method according to claim 1 or 2 wherein said image is classified as a far-shot if the size of said located face is substantially less than the size of said image.

4. A method according to claim 1 or 2 wherein said image is classified as a close-up where the size of said located face substantially corresponds with the size of said image

5. A method according to claim 1 or 2 wherein said image is classified as an extreme close-up where only a part of said located face appears within said image.

6. A method according to claim 1 or 2 wherein said classifying comprises associating a size of said located face with a set of predetermined thresholds for a size of a human face image.

7. A method according to claim 1 or 2 wherein said image is classified as a far shot if said image contains a face and the size of said located face is below a first predetermined threshold compared to the size of said image.

8. A method according to claim 7 wherein said image is classified as an extreme close up if the size of said located face is above a second predetermined threshold compared to the size of said image.

9. A method according to claim 8 wherein said image is classified as a close-up if the size of said located face is below said second predetermined threshold and above a third predetermined threshold compared to the size of said image.

10. A method according to claim 9 wherein said image is classified is a medium shot if the size of said located face is greater than said first predetermined threshold and less than said third predetermined threshold.

11. A method according to claim 1 wherein said analysing comprises interpreting information provided with said image.

12. A method according to claim 11 wherein said image comprises a frame of a digital video sequence of images.

13. A method according to claim 12 wherein said information is associated with other frames of said sequence.

14. A method according to claim 1 wherein said analysing comprises detecting one or more regions of said image at which skin coloured pixels are located in order to locate said face.

15. A method according to claim 1 wherein said determining approximates the size of said located face by a height and width of a bounding rectangle that encloses said face.

17. A method according to claim 16 wherein said image is classified using a term which provides information about an intention of a photographer whom captured said image.

18. A method according to claim 16 or 17 wherein said image is classified as a high-shot if the position of said located face is substantially toward a bottom of said image frame.

19. A method according to claim 16 or 17 wherein said image is classified as a eye-level shot where the position of said located face substantially corresponds with a centre of said image frame.

20. A method according to claim 16 or 17 wherein said image is classified as a low shot where the position of said located face is substantially toward a top of said image frame.

21. A method according to claim 16 or 17 wherein said image is classified as a left shot where the position of said located face is substantially toward a right hand side of said image frame.

22. A method according to claim 16 or 17 wherein said image is classified as a right shot where the position of said located face is substantially toward a left hand side of said image frame.

23. A method according to claim 16 or 17 wherein said image is classified as a low shot where the position of said located face is substantially toward a top of said image frame.

24. A method according to claim 16 wherein said analysing comprises interpreting information provided with said image.

25. A method according to claim 16 wherein said image comprises a frame of a digital video sequence of images.

26. A method according to claim 25 wherein said information is associated with other frames of said sequence.

27. A method according to claim 1 further comprising the steps of:

detecting an edge within said image;

determining an angle of inclination between said edge and an axis of said image frame;

classifying said image as a Dutch shot where said angle of inclination is between predetermined angles of inclination.

28. A method according to claim 27 wherein said predetermined angles of inclination comprise 30 and 60 degrees.

29. A method according to claim 16 further comprising:

analysing said image for the presence of a predetermined non-human component;

assessing said predetermined component with respect to at least one further criteria; and

where said criteria is met, classifying said image based upon the presence of said predetermined component.

30. A method according to claim 29 wherein said predetermined component comprises a colour of a distinct region of said image.

31. A method according to claim 29 wherein said criteria comprises at least a relative motion of said predetermined component within said image.

32. A method of processing an input sequence of images, said method comprising the steps of:

classifying each said image of said sequence using a method according to claim 1;

and

editing said sequence using said classification to form an output sequence of images.

33. A method according to claim 31 wherein said editing comprises applying an edit function to each said image of said input sequence, those ones of said images not satisfying said edit function being omitted from said output sequence.

34. A method according to claim 34 wherein said editing comprises establishing an editing template for said sequence, each said edit function forming a component of said template and corresponding to one of said image classifications.

35. A method according to claim 33 wherein said edit function comprises at least one effect for application to the image, said effect being selected from the group consisting of visual effects and audible effects.

36. A method according to claim 35 wherein said visual effects are selected from the group consisting of reproduction speed variation, zooming, blurring, and colour variation.

38. Apparatus according to claim 37 wherein:

(i) said image is classified as a far-shot if the size of said located face is substantially less than the size of said image;

(ii) said image is classified as a close-up where the size of said located face substantially corresponds with the size of said image; and

(iii) said image is classified as an extreme close-up where only a part of said located face appears within said image.

39. Apparatus according to claim 37 wherein said means for classifying associates a size of said located face with a set of predetermined thresholds for a size of a human face image.

40. Apparatus according to claim 39 wherein:

(i) said image is classified as a far shot if said image contains a face and the size of said located face is below a first predetermined threshold compared to the size of said image;

(ii) said image is classified as an extreme close up if the size of said located face is above a second predetermined threshold compared to the size of said image;

(iii) said image is classified as a close-up if the size of said located face is below said second predetermined threshold and above a third predetermined threshold compared to the size of said image; and

(iv) said image is classified is a medium shot if the size of said located face is greater than said first predetermined threshold and less than said third predetermined threshold.

41. Apparatus according to claim 37 wherein said analysing comprises interpreting information provided with said image.

42. Apparatus according to claim 41 wherein said image comprises a frame of a digital video sequence of images.

43. Apparatus according to claim 41 wherein said means for analysing detects one or more regions of said image at which skin coloured pixels are located in order to locate said face.

44. Apparatus according to claim 43 wherein said means for determining approximates the size of said located face by a height and width of a bounding rectangle that encloses said face.

46. Apparatus according to claim 45 wherein:

(i) said image is classified as a high-shot if the position of said located face is substantially toward a bottom of said image frame;

(ii) said image is classified as a eye-level shot where the position of said located face substantially corresponds with a centre of said image frame;

(iii) said image is classified as a low shot where the position of said located face is substantially toward a top of said image frame;

(iv) said image is classified as a left shot where the position of said located face is substantially toward a right hand side of said image frame;

(v) said image is classified as a right shot where the position of said located face is substantially toward a left hand side of said image frame;

(vi) said image is classified as a low shot where the position of said located face is substantially toward a top of said image frame.

47. Apparatus according to claim 46 wherein said analysing comprises interpreting information provided with said image.

48. Apparatus according to claim 46 wherein said image comprises a frame of a digital video sequence of images.

49. Apparatus according to claim 48 wherein said information is associated with other frames of said sequence.

50. Apparatus according to claim 37 further comprising:

means for detecting an edge within said image;

means for determining an angle of inclination between said edge and an axis of said image frame;

means for classifying said image as a Dutch shot where said angle of inclination is between predetermined angles of inclination.

51. Apparatus according to claim 37 further comprising:

means for analysing said image for the presence of a predetermined non-human component;

means for assessing said predetermined component with respect to at least one further criteria; and

where said criteria is met, classifying said image based upon the presence of said predetermined component.

52. Apparatus according to claim 51 wherein said predetermined component comprises a colour of a distinct region of said image.

53. Apparatus according to claim 51 wherein said criteria comprises at least a relative motion of said predetermined component within said image.

54. Apparatus for processing an image sequence, said apparatus comprising:

classification apparatus according to claim 37 for determining a classification for each image of said sequence; and

means for editing said sequence using said classification to form an output sequence of images.

55. Apparatus according to claim 54 wherein said means for editing comprises applying an edit function to each said image of said input sequence, those ones of said images not satisfying said edit function being omitted from said output sequence.

56. Apparatus according to claim 55 wherein said editing comprises establishing an editing template for said sequence, each said edit function forming a component of said template and corresponding to one of said image classifications.

57. Apparatus according to claim 56 wherein said edit function comprises at least one effect for application to the image, said effect being selected from the group consisting of visual effects and audible effects.

58. Apparatus according to claim 57 wherein said visual effects are selected from the group consisting of reproduction speed variation, zooming, blurring, and colour variation.

60. A computer readable medium according to claim 59 wherein:

(i) said image is classified as a far-shot if the size of said located face is substantially less than the size of said image;

(ii) said image is classified as a close-up where the size of said located face substantially corresponds with the size of said image; and

(iii) said image is classified as an extreme close-up where only a part of said located face appears within said image.

61. A computer readable medium according to claim 60 wherein said classifying comprises associating a size of said located face with a set of predetermined thresholds for a size of a human face image.

62. A computer readable medium according to claim 61 wherein:

(i) said image is classified as a far shot if said image contains a face and the size of said located face is below a first predetermined threshold compared to the size of said image;

(ii) said image is classified as an extreme close up if the size of said located face is above a second predetermined threshold compared to the size of said image;

(iii) said image is classified as a close-up if the size of said located face is below said second predetermined threshold and above a third predetermined threshold compared to the size of said image; and

(iv) said image is classified is a medium shot if the size of said located face is greater than said first predetermined threshold and less than said third predetermined threshold.

63. A computer readable medium according to claim 59 wherein said analysing comprises interpreting information provided with said image.

64. A computer readable medium according to claim 63 wherein said image comprises a frame of a digital video sequence of images.

65. A computer readable medium according to claim 64 wherein said information is associated with other frames of said sequence.

66. A computer readable medium according to claim 59 wherein said analysing comprises detecting one or more regions of said image at which skin coloured pixels are located in order to locate said face.

67. A computer readable medium according to claim 59 wherein said determining approximates the size of said located face by a height and width of a bounding rectangle that encloses said face.

68. A computer readable medium according to claim 59 further comprising:

code for analysing said image for the presence of a human face;

code for determining a position of the located face with respect to a frame of said image; and

code for classifying said image based on the relative position of said face with respect to said image frame.

69. A computer readable medium according to claim 68 wherein:

(i) said image is classified as a high-shot if the position of said located face is substantially toward a bottom of said image frame;

(ii) said image is classified as a eye-level shot where the position of said located face substantially corresponds with a centre of said image frame;

(iii) said image is classified as a low shot where the position of said located face is substantially toward a top of said image frame;

(iv) said image is classified as a left shot where the position of said located face is substantially toward a right hand side of said image frame;

(v) said image is classified as a right shot where the position of said located face is substantially toward a left hand side of said image frame;

(vi) said image is classified as a low shot where the position of said located face is substantially toward a top of said image frame.

70. A computer readable medium according to claim 69 wherein said analysing comprises interpreting information provided with said image.

71. A computer readable medium according to claim 69 wherein said image comprises a frame of a digital video sequence of images.

72. A computer readable medium according to claim 71 wherein said information is associated with other frames of said sequence.

73. A computer readable medium according to claim 72 further comprising:

code for detecting an edge within said image;

code for determining an angle of inclination between said edge and an axis of said image frame;

code for classifying said image as a Dutch shot where said angle of inclination is between predetermined angles of inclination.

74. A computer readable medium according to claim 73 wherein said predetermined angles of inclination comprise 30 and 60 degrees.

75. A computer readable medium according to claim 74 further comprising:

code for analysing said image for the presence of a predetermined non-human component;

code for assessing said predetermined component with respect to at least one further criteria; and

where said criteria is met, classifying said image based upon the presence of said predetermined component.

76. A computer readable medium according to claim 75 wherein said predetermined component comprises a colour of a distinct region of said image.

77. A computer readable medium according to claim 76 wherein said criteria comprises at least a relative motion of said predetermined component within said image.

78. A computer readable medium incorporating a computer program product for processing an input sequence of images, comprising:

code for classifying each said image of said sequence using the computer program product of claim 77; and

code for editing said sequence using said classification to form an output sequence of images.

79. A computer readable medium according to claim 78 wherein said editing comprises applying an edit function to each said image of said input sequence, those ones of said images not satisfying said edit function being omitted from said output sequence.

80. A computer readable medium according to claim 79 wherein said editing comprises establishing an editing template for said sequence, each said edit function forming a component of said template and corresponding to one of said image classifications.

81. A computer readable medium according to claim 80 wherein said edit function comprises at least one effect for application to the image, said effect being selected from the group consisting of visual effects and audible effects.

82. A computer readable medium according to claim 81 wherein said visual effects are selected from the group consisting of reproduction speed variation, zooming, blurring, and colour variation.

83. An edited sequence of images formed through implementation of a series of images according to any one of the preceding claims.

16. A method for automated classification of a digital image, said method comprising the steps of:

analysing said image for the presence of a human face;

determining a position of the located face with respect to a frame of said image; and

classifying said image based on the relative position of said face with respect to said image frame.

37. Apparatus for automated classification of a digital image, said comprising:

means for analysing said image for the presence of a human face;

means for determining a size of the located face with respect to a size of said image; and

means for classifying said image based on the relative size of said face with respect to said image.

45. Apparatus for automated classification of a digital image, said apparatus comprising:

means for analysing said image for the presence of a human face;

means for determining a position of the located face with respect to a frame of said image; and

means for classifying said image based on the relative position of said face with respect to said image frame.

59. A computer readable medium incorporating a computer program product operable upon computer apparatus for automated classification of a digital image, said computer program product comprising:

code for analysing said image for the presence of a human face;

code for determining a size of the located face with respect to a size of said image; and

code for classifying said image based on the relative size of said face with respect to said image.