Method, medium, and apparatus for detecting situation change of digital photo and method, medium, and apparatus for situation-based photo clustering in digital photo album

• US 20050280719A1
• Filed: 04/20/2005
• Published: 12/22/2005
• Est. Priority Date: 04/21/2004
• Status: Abandoned Application

• Current Assignee: Research and Industrial Cooperation Group, Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Yang, Seungil, Kim, Sangkyun, Moon, Youngsu, Ro, Yongman, Kim, Jlyeun
• Application Number:
  US11/109,859
• Publication Number:
  US 20050280719A1
• Time in Patent Office: Days
• US Class Current: 348/231.2
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location

Abstract

An apparatus, medium, and method for detecting a situation change of a digital photo, and a method, medium, and apparatus for situation-based clustering in a digital photo album. The apparatus for situation-based clustering in a digital photo album includes: a photo description information generation unit generating photo description information describing a photo and including at least a photo descriptor; an albuming tool description information generation unit generating albuming tool description information including a predetermined parameter for situation-based clustering of digital photos; an albuming tool performing photo albuming through situation-based clustering by using at least the photo description information and the albuming tool description information; a photo group information generation unit generating predetermined photo group description information from the output of the albuming tool; and a photo albuming information generation unit generating photo albuming information by using photo description information and photo group description information.

Claims

1. An apparatus for detecting a situation change in digital photos, comprising:

a photo sort unit sorting photos, desired to be situation-based clustered, in order of time;

a time feature value obtaining unit obtaining predetermined time feature values from each of two contiguous photos among the sorted photos;

a content-based feature value extraction unit extracting predetermined content-based feature values from each of the two contiguous photos;

a dissimilarity measuring unit measuring dissimilarity between the two photos by making predetermined time feature value importances reflect respective time feature values, and by making predetermined content-based feature value importances reflect respective content-based feature values; and

a situation change detection unit detecting a situation change by determining the situation change if an amount of the dissimilarity is equal to or greater than a predetermined threshold.

2. The apparatus of claim 1, wherein a predetermined content-based feature value is generated based on pixel information of a photo, and includes a visual descriptor including color, texture, and shape feature values, and an audio descriptor including a voice feature value.

3. The apparatus of claim 1, wherein a time feature value importance and a content-based feature value importance are determined by referring to situation-based clustering hints including at least one of entire brightness information of a photo (Brightness), complexity information of the photo (Level of detail), homogeneous texture information of the photo (Homogeneous texture), edge information of the photo (Heterogeneous texture), information on whether the photo is monochrome (Monochromatic), information indicating a degree of colorfulness of a color expression of the photo (colorfulness), information indicating an entire color coherence shown in the photo (color coherence), information indicating a color temperature of a color of the photo (color temperature), information indicating whether a photo file of the photo includes taken time information (Taken time), information indicating that, if the photo and another photo are taken by different cameras in similar time bands and are clustered together, time information of the photo overlaps time information of the other photo and an importance of corresponding time information is lowered when the photo is situation-based clustered (Time overlap), information indicating whether voice information of a user is stored together with the photo when the photo was taken and is included with the photo as an audio clip file (Audio clip), and information indicating voice words and sentence strings recognized in an audio file of the photo (Speech recognition).

4. An apparatus for situation-based clustering of a digital photo album, comprising:

a photo description information generation unit generating photo description information describing a photo and including at least a photo identifier;

an albuming tool description information generation unit generating albuming tool description information including a predetermined parameter for situation-based clustering of digital photos;

an albuming tool performing photo albuming through situation-based clustering by using at least the photo description information and the albuming tool description information;

a photo group information generation unit generating predetermined photo group description information from an output of the albuming tool; and

a photo albuming information generation unit generating predetermined photo albuming information by using the photo description information and the predetermined photo group description information for situation-based clustering of the digital photo album.

5. The apparatus of claim 4, wherein among the photo identifier, information on an author of the photo, photo file information, camera information, photographing information, and a content-based feature value, the photo description information includes at least the photo identifier, with the content-based feature value being generated by using pixel information of the photo, and includes a visual descriptor including color, texture, and shape feature values, and/or an audio descriptor including a voice feature value.

6. The apparatus of claim 4, wherein the albuming tool description information generation unit comprises at least one of:

a sort key generation unit generating items for sorting photos before clustering the photos;

a situation-based clustering hint generation unit generating a situation-based clustering hint to help photo clustering; and

an importance generation unit generating importances of information to be used in photo clustering.

7. The apparatus of claim 6, wherein the photo sort items of the sort key generation unit include at least one of a file name, a photographing time, and a photo file creation time.

8. The apparatus of claim 6, wherein the photographing time includes photographing date information and the photo file creation time includes photo file creation date information.

9. The apparatus of claim 6, wherein the situation-based clustering hint of the situation-based clustering hint unit includes at least one of entire brightness information of the photo (Brightness), complexity information of the photo (Level of detail), homogeneous texture information of the photo (Homogeneous texture), edge information of the photo (Heterogeneous texture), information on whether the photo is monochrome (Monochromatic), information indicating a degree of colorfulness of a color expression of the photo (colorfulness), information indicating an entire color coherence shown in the photo (color coherence), information indicating a color temperature of a color of the photo (color temperature), information indicating whether a photo file of the photo includes taken time information (Taken time), information indicating that, if the photo and another photo are taken by different cameras in similar time bands and are clustered together, time information of the photo overlaps time information of the other photo and an importances of corresponding time information is lowered when the photo is situation-based clustered (Time overlap), information indicating whether voice information of a user is stored together with the photo when the photo was taken and is included with the photo as an audio clip file (Audio clip), and information indicating voice words and sentence strings recognized in an audio file of the photo (Speech recognition).

10. The apparatus of claim 6, wherein the importances of the importance generation unit are based on at least one of:

information (taken time) setting an importance of time information on a time when the photo is taken; and

information (low-level feature) setting an importance of content-based feature value information of the photo.

11. The apparatus of claim 10, wherein the information (low-level feature) setting the importance of content-based feature value information of the photo comprises:

information setting an importance of a moving picture experts group (MPEG)-7 Visual Descriptor; and

information setting an importance of a MPEG-7 Audio Descriptor.

12. The apparatus of claim 4, wherein the albuming tool comprises:

a situation-based photo clustering tool clustering digital photo data based on situations.

13. The apparatus of claim 12, wherein the situation-based photo clustering tool comprises:

a photo sort unit sorting photos, desired to be situation-based clustered, in order of time;

a time feature value obtaining unit obtaining, from the photo description information generation unit, time feature values from each of two contiguous photos among the sorted photos;

a content-based feature value extraction unit extracting, from the photo description information generation unit, content-based feature values from each of the two contiguous photos;

a dissimilarity measuring unit measuring dissimilarity between the two photos by making time feature value importances, obtained from the albuming tool description information generation unit, reflect respective time feature values obtained from the time feature value obtaining unit, and by making predetermined content-based feature value importances, obtained from the albuming tool description information generation unit, reflect respective content-based feature values extracted in the content-based feature value extraction unit; and

a situation change detection unit detecting a situation change by determining the situation change based on an amount of the dissimilarity value.

14. The apparatus of claim 13, wherein the respective time feature value importances and the respective predetermined content-based feature value importances are determined by referring to situation-based clustering hints of the albuming tool description information generation unit.

15. The apparatus of claim 4, wherein the photo group description information of the photo group information generation unit includes at least one of situation-based photo groups by clustering situation-based photos, and a situation-based photo group comprises a situation identifier identifying a situation, a series of photos formed with a plurality of photos determined by photo identifiers, and a photo key identifier allowing identifying of one or more representative photos among photos in a photo group.

16. A method for detecting a situation change in digital photos, comprising:

sorting photos, desired to be situation-based clustered, in order of time;

obtaining respective time feature values and respective predetermined content-based feature values from each of two contiguous photos among the sorted photos;

measuring a dissimilarity between the two photos by making predetermined time feature value importances reflect respective time feature values, and by making predetermined content-based feature value importances reflect respective content-based feature values; and

detecting a situation change by determining the situation change if an amount of the dissimilarity is equal to or greater than a predetermined threshold.

17. The method of claim 16, wherein the detecting of the situation change comprises determining the situation change if an amount of change, between a dissimilarity between one of the contiguous photos and a previous photo, not same as another one of the contiguous photos, and a dissimilarity between the other one of the contiguous photos and a subsequent photo, is greater than a threshold.

18. The method of claim 16, wherein a predetermined content-based feature value is generated by using pixel information of respective photos, and includes a visual descriptor including color, texture, and shape feature values, and/or an audio descriptor including a voice feature value.

19. The method of claim 16, wherein a time feature value importance and a content-based feature value importance are determined by referring to a situation-based clustering hint including at least one of entire brightness information of a photo (Brightness), a complexity information of the photo (Level of detail), homogeneous texture information of the photo (Homogeneous texture), edge information of the photo (Heterogeneous texture), information on whether the photo is monochrome (Monochromatic), information indicating a degree of colorfulness of a color expression of the photo (colorfulness), information indicating entire color coherence shown in the photo (color coherence), information indicating a color temperature of a color of the photo (color temperature), information indicating whether a photo file of the photo includes taken time information (Taken time), information indicating that, if the photo and another photo taken by different cameras in similar time bands and are clustered together, time information of the photo overlaps time information of the other photo and an importance of corresponding time information is lowered when the photo is situation-based clustered (Time overlap), information indicating whether voice information of a user is stored together with the photo when the photo was taken and is included as an audio clip file (Audio clip), and information indicating voice words and sentence strings recognized in an audio file of the photo (Speech recognition).

30. A medium comprising computer readable instructions implementing the method of claim 16.

20. A method for situation-based clustering of a digital photo album, comprising:

generating photo description information by extracting at least one of camera information on a camera taking a photo, photographing information of the photo, and a content-based feature value of the photo;

generating albuming tool description information including a predetermined parameter for situation-based clustering of digital photos;

performing photo albuming through situation-based clustering by using at least the photo description information and the albuming tool description information;

generating photo group description information by using a result of the situation-based clustering; and

generating predetermined photo albuming information by using the photo description information and the photo group description information to situation-based cluster the digital photo album.

21. The method of claim 20, wherein in the generating of the photo description information, among the photo identifier, information on an author of the photo, photo file information, camera information, photographing information, and content-based feature values, the photo description information includes at least the photo identifier, with the content-based feature value being generated by using pixel information of the photo, and includes a visual descriptor including color, texture, and shape feature values, and/or an audio descriptor including a voice feature value.

22. The method of claim 20, wherein in the generating of the albuming tool description information, the albuming tool description information generation comprises at least one of:

sort key sorting photos before clustering of the photos;

situation-based clustering hint generating a situation-based clustering hint to help photo clustering; and

importance generating importances of information to be used in photo clustering.

23. The method of claim 22, wherein a sort key in the sort key sorting includes at least one of a file name, a photographing time, and a photo file creation time.

24. The method of claim 22, wherein the situation-based clustering hint includes at least one of entire brightness information of the photo (Brightness), complexity information of the photo (Level of detail), homogeneous texture information of the photo (Homogeneous texture), edge information of the photo (Heterogeneous texture), information on whether the photo is monochrome (Monochromatic), information indicating a degree of colorfulness of a color expression of the photo (colorfulness), information indicating an entire color coherence shown in the photo (color coherence), information indicating a color temperature of a color of the photo (color temperature), information indicating whether a photo file of the photo includes taken time information (Taken time), information indicating that, if the photo and another photo taken by different cameras in similar time bands and are clustered together, time information of the photo overlaps time information of the other photo and an importance of corresponding time information is lowered when the photo is situation-based clustered (Time overlap), information indicating whether voice information of a user stored together with the photo when the photo was taken and is included with the photo as an audio clip file (Audio clip), and information indicating voice words and sentence strings recognized in an audio file of the photo (Speech recognition).

25. The method of claim 22, wherein the importances are based on at least one of:

information (taken time) setting an importance of time information on a time when the photo is taken; and

information (low-level feature) setting an importance of content-based feature value information of the photo.

26. The method of claim 25, wherein the information (low-level feature) setting the importance of content-based feature value information of the photo comprises:

information setting an importance of a MPEG-7 Visual Descriptor; and

information setting an importance of a MPEG-7 Audio Descriptor.

27. The method of claim 20, wherein the performing of the photo albuming comprises:

sorting photos, desired to be situation-based clustered, in order of time;

obtaining time feature values and predetermined content-based feature values from each of two contiguous photos among the sorted photos;

measuring a dissimilarity between the two photos by making predetermined time feature value importances reflect respective time feature values, and by making predetermined content-based feature value importances reflect respective content-based feature values; and

detecting a situation change by determining the situation change based on an amount of the dissimilarity value.

28. The method of claim 27, wherein the time feature value importances and the content-based feature value importances are determined by referring to situation-based clustering hints.

29. The method of claim 20, wherein in the generating of the predetermined photo albuming information, the photo group description information includes at least one of situation-based photo groups by clustering situation-based photos, and with the situation-based photo group comprising a situation identifier identifying a situation, a series of photos formed with a plurality of photos determined by photo identifiers, and a photo key identifier allowing identifying of one or more representative photos among the photos in a photo group.

31. A medium comprising computer readable instructions implementing the method of claim 20.