Method and apparatus for category-based photo clustering in digital photo album

• US 20060074771A1
• Filed: 10/04/2005
• Published: 04/06/2006
• Est. Priority Date: 10/04/2004
• Status: Abandoned Application

• Current Assignee: Research and Industrial Cooperation Group, Samsung Electronics Co. Ltd.
• Original Assignee: Research and Industrial Cooperation Group, Samsung Electronics Co. Ltd.
• Inventors: Kim, Sangkyun, Moon, Youngsu, Kim, Jiyeun, Yang, Seungji, Ro, Yongman
• Application Number:
  US11/242,533
• Publication Number:
  US 20060074771A1
• Time in Patent Office: Days
• US Class Current: 705/26
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

A method of category-based clustering of a digital photo album and a system thereof, the method includes: generating photo information by extracting at least one of camera information of a camera used to take a photo, photographing information, and a content-based feature value including at least one of color, texture, and shape feature values, and a speech feature value; generating a predetermined parameter including at least one of user preference indicating the personal preference of the user, photo semantic information generated by using the content-based feature value of the photo, and photo syntactic information generated by at least one of the camera information, the photographing information, and interaction with the user; generating photo group information categorizing photos by using the photo information and the parameter; and generating a photo album by using the photo information and the photo group information. According to the method and system, by using together user preference and content-based feature value information, such as color, texture, and shape, from the contents of photos, as well as information that can be basically obtained from photos, such as camera information and file information stored in a camera, a large volume of photos are effectively categorized such that an album can be fast and effectively generated with photo data.

Claims

1. A method of category-based clustering in a digital photo album, comprising:

generating photo information by extracting at least one of camera information of a camera used to take a photo, photographing information, and a content-based feature value of the photo including at least one of color, texture, and shape feature values, a speech feature value, or combinations thereof;

generating a predetermined parameter including at least one of user preference indicating a personal preference of the user, photo semantic information generated by using the content-based feature value of the photo, photo syntactic information or combinations thereof, with the photo syntactic information being generated by at least one of the camera information, the photographing information, interaction with the user or combinations thereof;

generating photo group information categorizing photos by using the photo information and the predetermined parameter; and

generating a photo album by using the photo information and the photo group information.

28. A computer readable recording medium having embodied thereon a computer program for executing the method of claim 1.

2. A method of category-based clustering in a digital photo album, comprising:

generating photo description information describing a photo and including at least a photo identifier;

generating albuming tool information supporting photo categorization and including at least a predetermined parameter for photo categorization;

categorizing photos by using input photos, the photo description information and the albuming tool information;

generating the categorized result as predetermined photo group description information; and

generating predetermined photo album information by using the photo description information and the predetermined photo group description information.

3. The method of claim 2, wherein the generating of the photo description information comprises:

extracting camera information of a camera used to take the photo and photographing information from a photo file;

extracting a content-based feature value from pixel information of the photo; and

generating photo description information by using the extracted camera information, photographing information and content-based feature value, and

the content-based feature value comprises;

a visual descriptor including color, texture, and shape feature values; and

an audio descriptor including a speech feature value, and

the photo description information comprises at least the photo identifier, information of a photographer taking the photo, photo file information, the camera information, the photographing information, and the content-based feature value.

4. The method of claim 3, wherein the photo file information comprises at least one of a file name, file format, file size, file creation date, or combinations thereof, and

the camera information comprises at least one of information (IsEXIFInformation) indicating whether or not the photo file includes EXIF information, information (Camera model) indicating a camera model used to take the photo, or combinations thereof, and

the photographing information comprises at least one of information (Taken date/time) indicating a date and time when the photo is taken, information (GPS information) indicating a location where the photo is taken, photo width information (Image width), photo height information (Image height), information (Flash on/off) indicating whether or not a camera flash is used to take the photo, brightness information of the photo (Brightness), contrast information of the photo (Contrast), sharpness information of the photo (Sharpness), or combinations thereof.

5. The method of claim 3, wherein in the generating of the albuming tool information, the albuming tool description information comprises at least one of:

a category list indicating semantic information to be categorized;

a category-based clustering hint to help photo clustering, or combinations thereof, and

the category-based clustering hint comprises at least one of;

a semantic hint generated by using the content-based feature value of the photo;

a syntactic hint generated by at least one of the camera information, the photographing information and interaction with a user;

a user preference hint, or combinations thereof.

6. The method of claim 5, wherein the category list comprises at least one of mountain, waterside, human-being, indoor, building, animal, plant, transportation, object, or combinations thereof.

7. The method of claim 5, wherein the semantic hint is semantic information included in the photo, the information expressed by using nouns, adjectives, and adverbs.

8. The method of claim 5, wherein the syntactic hint comprises at least one of:

a camera hint indicating the camera information at the time of photographing;

an image hint including at least one of information (Photographic composition) on a composition formed by objects of the photo, information (Region of interest) of a number of main interest areas in the photo and a location of each area, a relative compression ratio (Relative compression ratio) in relation to the resolution of the photo, or combinations thereof;

an audio hint including keywords (Speech info) describing speech information extracted from an audio clip, or combinations thereof.

9. The method of claim 8, wherein the camera hint is based on EXIF information stored in a photo file and comprises at least one of a photographing time (Taken time), information (Flash info) on whether or not a flash is used, information (Zoom info) on whether or not a camera zoom is used and the zoom distance, a camera focal length (Focal length), a focused region (Focused region), an exposure time (Exposure time), information (Contrast) on contrast basically set for the camera, information (Brightness) on brightness basically set for the camera, GPS information (GPS info), text annotation information (Annotation), camera angle information (Angle), or combinations thereof.

10. The method of claim 5, wherein the user preference hint comprises:

category preference information (Category preference) describing a preference of the user on categories in the category list.

11. The method of claim 5, wherein the categorizing of the photos comprises:

generating a new feature value by applying the category-based clustering hint to the extracted content-based feature value;

measuring similarity distance values between the new feature value and feature values in a predetermined category feature value database; and

determining one or more categories satisfying a condition that a similarity distance value is less than a predetermined threshold, as final categories.

12. The method of claim 11, wherein the semantic hint, the syntactic hint and the user preference hint values are extracted and a value of the category-based clustering hint is expressed as the following equation:

V_hint(i)={V_semantic(i), V_syntactic(i), V_user}

where V_semantic(i) denotes a semantic hint extracted from the i-th photo, V_syntactic(i) denotes a syntactic hint extracted from the i-th photo, and V_user denotes a user category preference hint.

13. The method of claim 12, wherein in the user preference hint value extraction, a category on which sets of input query photo data belong is selected according to a memory of the user, an importance degree of each category is input, and the category preference hint of the user is expressed as the following equation:

V_user={β

₁,β

₂,β

₃, . . . ,β

_c, . . . ,β

_C}

where β

_c is a value denoting the preference degree of the user on a c-th category and has a value between 0.0 to 1.0 inclusive, and a method of selecting a category by the above equation is expressed as the following equation;

S_category^selected={β

₁S₁,β

₂S₂,β

₃S₃, . . . ,β

_cS_c, . . . ,β

_CS_C}

where S_c denotes the c-th category, and if β

_c is 0.0, the category is not selected, and if β

_c is close to 0.0, the category is selected but indicates the user preference of the category is low, and if β

_c is close to 1.0, β

_c indicates that the user preference of the selected category is high.

14. The method of claim 12, wherein in the extraction of the syntactic hint value, by using EXIF information, image composition information, and audio clip information stored in the camera, the semantic hint value is extracted and the semantic hit value extracted from an i-th photo is expressed as the following equation:

V_syntactic(i)={V_camera, V_image, V_audio}

where V_camera denotes a set of syntactic hints including camera information and photographing information, V_image denotes a set of syntactic hints extracted from photo data itself, and V_audio denotes a set of syntactic hint values extracted from an audio clip stored together with photos.

15. The method of claim 12, wherein in the extraction of the semantic hint value, a semantic hint value included in the contents of the photo is extracted in a j-th area of the i-th photo, and is expressed as the following equation:

V_semantic(i,j)={V₁, V₂, V₃, . . . , V_M} where V_m=(ν

_m^adverb, ν

_m^adjective, ν

_m^noun, α

_m)

where V_m denotes an m-th semantic hint value extracted in the j-th area of the i-th photo, ν

_m^noun denotes the m-th noun hint value, ν

_m^adverb denotes the m-th adverb hint value, ν

_m^adjective denotes the m-th adjective hint value, and α

_m denotes a value indicating the importance of the m-th semantic hint value, and has a value between 0.0 and 1.0 inclusive.

16. The method of claim 11, wherein in relation to the content-based feature value, by using extracted category hint information items, an image is localized and from each area, multiple content-based feature values are extracted and multiple content-based feature values in a j-th area of the i-th photo are expressed as the following equation:

F_content(i, j)={F₁(i, j), F₂(i, j), F₃(i, j), . . . , F_N(i, j)}

where F_k(i,j) denotes a k-th feature value vector in the j-th area of the i-th photo.

17. The method of claim 11, wherein in the generating of the new feature value, the new feature value is expressed as the following equation:

F_combined(i)=Φ

{V_hint(i), F_content(i)}

where function Φ

(·

) is a function generating a feature value by using together V_hint(i), the category-based clustering hint of the i-th photo, and F_content(i), the content-based feature value of the i-th photo, and

in the measuring of the similarity distance value, the similarity distance value is expressed as the following equation;

D(i)={D₁(i), D₂(i), D₃(i), . . . , D_c(i)}

where D_c(i) denotes the similarity distance value between the c-th category and the i-th photo, and

in the determining one or more categories, the condition is expressed as the following equation;

S_target(i) ⊂

{S₁,S₂,S₃, . . . ,S_C}, subject to D_Sc(i)≦

th_D

where {S₁, S₂, S₃, . . . , S_c} denotes a set of categories, th_D denotes a threshold of a similarity distance value for determining a category, and S_target(i) denotes a set of categories satisfying the condition and indicates the category of the i-th photo.

18. The method of claim 3, wherein in the generating of the categorized result as the predetermined photo group description information, the photo group description information comprises:

a category identifier generated by referring to the category list; and

a series of photos formed with a plurality of photos determined by the photo identifier.

29. A computer readable recording medium having embodied thereon a computer program for executing the method of claims 2.

19. An apparatus for category-based clustering in a digital photo album, comprising:

a photo description information generation unit generating photo description information describing a photo and including at least a photo identifier;

an albuming tool description information generation unit generating albuming tool description information supporting photo categorization and including at least a predetermined parameter for the photo categorization;

an albuming tool performing photo albuming including the photo categorization by using at least the photo description information and the albuming tool description information;

a photo group information generation unit generating photo group description information from the photo albuming; and

a photo album information generation unit generating predetermined album information by using the photo description information and the photo group description information.

20. The apparatus of claim 19, wherein the photo description information comprises at least one of a photo identifier among the photo identifier, information on a photographer taking the photo, photo file information, camera information, photographing information, content-based feature value, or combinations thereof, and

the content-based feature value is generated by using pixel information of the photo and comprises:

a visual descriptor including color, texture, and shape feature values; and

an audio descriptor including a speech feature value.

21. The apparatus of claim 19, wherein the albuming tool description information generation unit comprises at least one of:

a category list generation unit generating a category list indicating semantic information to be categorized;

a clustering hint generation unit generating a category-based clustering hint to help photo clustering, or combinations thereof, and

the clustering hint generation unit comprises at least one of;

a semantic hint generation unit generating a semantic hint by using the content-based feature value of the photo;

a syntactic hint generation unit generating a syntactic hint by at least one of the camera information, the photographing information and interaction with a user;

a preference hint generation unit generating a preference hint of the user, or combinations thereof.

22. The apparatus of claim 21, wherein the category list of the category list generation unit comprises at least one of mountain, waterside, human-being, indoor, building, animal, plant, transportation, and object.

23. The apparatus of claim 21, wherein the semantic hint of the semantic hint generation unit is semantic information included in the photo, the semantic information expressed by using nouns, adjectives, and adverbs.

24. The apparatus of claim 21, wherein the syntactic hint of the syntactic hint generation unit comprises at least one of:

a camera hint indicating the camera information at time of photographing;

an image hint including at least one of information (Photographic composition) on a composition formed by objects of the photo, information (Region of interest) on a number of main interest areas in the photo and a location of each main interest area, and a relative compression ratio (Relative compression ratio) in relation to a resolution of the photo; and

an audio hint including keywords (Speech info) describing speech information extracted from an audio clip.

25. The apparatus of claim 19, wherein the albuming tool comprises a category-based photo clustering tool clustering digital photo data based on the category.

26. The apparatus of claim 25, wherein the category-based photo clustering tool comprises:

a feature value generation unit generating a new feature value, by using content-based feature value generated in the photo description information generation unit and category-based clustering hint generated in the albuming tool description information generation unit;

a feature value database extracting in advance and storing feature values of photos belonging to a category;

a similarity measuring unit measuring similarity distance values between a new feature value and feature values in the feature value database; and

a category determination unit determining one or more categories satisfying a condition that the similarity distance value is less than a predetermined threshold, as final categories.

27. The apparatus of claim 19, wherein the photo group description information of the photo group information generation unit comprises:

a category identifier generated by referring to a category list; and

a series of photos formed with a plurality of photos determined by the photo identifier.

35. A camera comprising the apparatus of claim 19.

30. A method of category-based clustering in a digital photo album, comprising:

generating photo description information describing the photo and including at least a photo identifier;

generating albuming tool description information supporting photo categorization and including at least a predetermined parameter for photo categorization;

categorizing the photo using the photo description information and the albuming tool description information;

generating photo group description information from the categorized photo; and

generating predetermined photo album information using the photo description information and the photo group description information.

31. The method of claim 30, wherein the photo description information is generated by extracting camera information, and photographing information from a photo file and by extracting a content-based feature value from pixel information of the photo.

32. The method of claim 31, wherein the content-based feature value includes a visual descriptor including color, texture, and shape feature values, and an audio descriptor including a speech feature value.

33. The method of claim 30, wherein the photo description information includes the photo identifier, photographer information, photo file information, camera information, photographing information and content-based feature value.

34. The method of claim 31, wherein the categorization of the photo includes:

generating a new feature value by applying a category-based clustering hint to the extracted content-based feature value;

measuring similarity distance values between the new feature value and feature values in a predetermined category feature value database; and

determining as final categories one or more categories satisfying a condition that the similarity distance value is less than a predetermined threshold.