Method and system for albuming multimedia using albuming hints

• US 20060239591A1
• Filed: 04/18/2006
• Published: 10/26/2006
• Est. Priority Date: 04/18/2005
• Status: Abandoned Application

• Current Assignee: Research and Industrial Cooperation Group, Samsung Electronics Co. Ltd.
• Original Assignee: Research and Industrial Cooperation Group, Samsung Electronics Co. Ltd.
• Inventors: Kim, Sangkyun, Kim, Jiyeun, Yang, Seungji, Ro, Yongman
• Application Number:
  US11/405,566
• Publication Number:
  US 20060239591A1
• Time in Patent Office: Days
• US Class Current: 382/305
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

A method and apparatus album multimedia using media albuming hints. The multimedia albuming method includes: extracting albuming hints from multimedia contents; describing the extracted albuming hint information in a predetermined description structure; generating a media descriptor by using the described albuming hint information; and albuming multimedia contents by using the media descriptor. According to the method and apparatus, digital multimedia, such as digital photos, music, and video data (moving pictures), may be albumed automatically or semiautomatically. Also, media albuming hints included in the present method and apparatus may be used such that the performance of albuming functions, such as indexing or clustering with semantic information of multimedia contents, may be enhanced. Furthermore, by reducing the complexity of calculations required for albuming, the albuming may be performed more efficiently.

Claims

1. A multimedia albuming method comprising:

extracting albuming hints information from multimedia contents;

describing the extracted albuming hints information in a predetermined description structure;

generating a media descriptor by using the described albuming hint information; and

albuming multimedia contents by using the media descriptor.

3. The method of claim 1, further comprising:

obtaining contents from a multimedia content acquisition apparatus and performing preprocessing; and

receiving inputs of the multimedia contents and the metadata corresponding to the multimedia contents obtained from the multimedia content acquisition apparatus.

4. The method of claim 1, wherein the albuming hint information comprises photo albuming hint information, music albuming hint information and video albuming hint information.

5. The method of claim 4, wherein the description structure of the photo albuming hint information comprises a description structure expressing information on a time when a photo is taken and camera information, a description structure expressing a perceptual characteristic of human beings with respect to contents of the photo, a description structure expressing information on a person included in the photo, a description structure expressing information on a view of the photo, and a description structure expressing information on a popularity of the photo.

6. The method of claim 5, wherein the description structure expressing information on a time when the photo is taken and camera information comprises at least one of information indicating whether photo data includes Exif information as metadata, photographer information, photographing time information, manufacturer information on a manufacturer of a camera with which the photo is taken, camera model information on a model of the camera with which a photo is taken, shutter speed information on a shutter speed when the photo is taken, color mode information on a color mode when the photo is taken, information indicating sensitivity of film when the photo is taken, information indicating whether a flash is used when the photo is taken, information indicating a degree of opening of an iris of a camera lens when the photo is taken, information indicating a distance of an optical zoom which is used when the photo is taken, information indicating a focal length when the photo is taken, information indicating a distance between a focused object and a camera when the photo is taken, GPS information in relation to a place where the photo is taken, information indicating a direction in which a first pixel of a photo image is located, as a direction of a camera when the photo is taken, information indicating sound recorded together when the photo is taken, and information indicating a thumbnail image stored for high-speed browsing in a camera after the photo is taken.

7. The method of claim 5, wherein the description structure expressing the perceptual characteristic of human beings with respect to the contents of a photo comprises at least one of an item (avgColorfulness) indicating a degree of colorful expression of the photo, an item (avgColorCoherence) indicating a degree of coherence of an entire color expressed in the photo, an item (avgLevelOfDetail) indicating a precision of the contents included in the photo, an item (avgHomogenity) indicating homogeneity of texture information of the contents of the photo, an item (avgPowerOfEdge) indicating a robustness of edge information of the contents included in the photo, an item (avgDepthOfField) indicating a depth of a focus of a camera with respect to the contents included in the photo, an item (avgBlurness) indicating a degree of blur of the contents of the photo by a shake occurring when a camera shutter is pressed, an item (avgGlareness) indicating a degree that the contents of the photo are hidden by light when a large quantity of flash light is used to take the photo or an external light source with a large quantity of strong light is used, and an item (avgBrightness) indicating an entire brightness of the photo.

8. The method of claim 7, wherein the item (avgColorfulness) indicating the degree of colorful expression of the photo is measured by normalizing a height of a histogram of each RGB color value from a color histogram and a distribution value of an entire color value, or by using a distribution value of colors measured by using CIE L*u*v* color space.

9. The method of claim 7, wherein the item (avgColorCoherence) indicating the degree of coherence of the color expressed in the photo is measured by using a Dominant Color descriptor selected from the group consisting of MPEG-7 visual descriptors, and is measured by normalizing a histogram height of each color value from a color histogram and a distribution value of an entire color value.

10. The method of claim 7, wherein the item (avgLevelOfDetail) indicating the precision of the contents included in the photo is measured by using entropy measured from pixel information of the photo, or by using an isopreference curve that is an element to determine an actual complexity of the photo, or by a relative measuring method in which compression ratios are compared with each other when compression is performed under substantially identical conditions.

11. The method of claim 7, wherein the item (avgHomogeneity) indicating homogeneity of texture information of the contents of the photo is measured using regularity, direction and scale of texture from feature values of a Texture Browsing descriptor selected from the group consisting of MPEG-7 visual descriptors.

12. The method of claim 7, wherein the item (avgPowerOfEdge) indicating the robustness of edge information of the contents included in the photo is measured by extracting edge information from the photo and normalizing a strength of an extracted edge.

13. The method of claim 7, wherein the item (avgDepthOfField) indicating the depth of the focus of the camera with respect to the contents included in the photo is measured generally by using a focal length of a camera lens, a diameter of the camera lens, and figures of an iris.

14. The method of claim 7, wherein the item (avgBlurness) indicating the degree of blur of the contents of the photo by a shake occurring when a camera shutter is pressed is measured using a power of an edge of the contents of the photo.

15. The method of claim 7, wherein the item (avgGlareness) indicating the degree that the contents of the photo are hidden by an external light source with a large quantity of strong light is measured by using a brightness of a photo pixel value.

16. The method of claim 7, wherein the item (avgBrightness) indicating the entire brightness of the photo is measured using a brightness of a photo pixel value.

17. The method of claim 5, wherein the description structure expressing information on the person included in the photo comprises an item indicating a number of persons included in the photo, an item indicating position information on a position of a face of each person and a position of clothes worn by the person, and an item indicating relationships among persons included in the photo.

18. The method of claim 17, wherein the item indicating position information on the position of the face of each person and the position of the clothes worn by the person comprises an identification of the person, and the position of the clothes worn by the person.

19. The method of claim 17, wherein the item indicating the relationships among persons included in the photo comprises an item indicating a first person of two persons whose relationship is to be indicated, an item indicating a second person of the two persons whose relationship is to be indicated, and an item indicating a relationship between the two persons.

20. The method of claim 5, wherein the description structure expressing information on the view of the photo comprises an item indicating whether a major part shown in the photo is a background or a foreground, an item indicating a position of a part corresponding to the background in the contents expressed in the photo, and an item indicating a position of a part corresponding to the foreground in the contents expressed in the photo.

21. The method of claim 4, wherein the description structure of the music albuming hint information comprises at least one of a description structure expressing information on a time when a music file is recorded, generated or edited, a description structure expressing a part that is a highlight of a music file, a description structure expressing a level of perceptual sound quality of a music file, a description structure expressing information on a mood of music, a description structure expressing information on a situation suitable to reproduce a music file, a description structure expressing media resource information on photos or moving pictures related to a music file, and a description structure expressing popularity or preference of a music file.

22. The method of claim 21, wherein in case of an MP3 file, the description structure expressing information on a time when music is recorded, generated or edited comprises at least one of a description structure indicating whether metadata in relation to a music file includes ID3 header information, a description structure indicating a title of a music file, a description structure indicating a name of a singer or player of music, a description structure indicating a genre of music, a description structure indicating a total reproduction time of a music file, a description structure indicating information on lyrics of music, and a description structure indicating a language of a music file.

23. The method of claim 4, wherein the description structure of the video albuming hint information comprises a description structure expressing information on major characters included in a video file, a description structure expressing a part that is a highlight of a video file, and a description structure expressing a popularity or preference of a video file.

24. The method of claim 1, wherein the described albuming hint information is used by a media description tool to generate a media descriptor that is metadata to describe media together with content-based feature value metadata.

25. The method of claim 1, wherein in the albuming of the multimedia contents, at least one of photo data, music data and video data is clustered or indexed using the media descriptor.

26. The method of claim 25, wherein the clustering or indexing of the photo data comprises at least one of:

albuming photos based on a situation in which a photo is taken;

albuming photos based on a semantic category included in the photo; and

albuming photos based on a person included in the photo.

27. The method of claim 25, wherein the clustering or indexing of the music data comprises at least one of:

albuming music based on ID3 metadata; and

albuming music based on a mood of a music file.

28. The method of claim 25, wherein the clustering or indexing of the video data comprises at least one of:

albuming video data based on a basic unit shot of a video segment;

albuming video data based on a scene having semantic information in addition to a shot;

albuming video data based on a genre of a video file; and

albuming based on a person included in the video file

29. The method of claim 1, wherein the albuming of the multimedia contents comprises at least one of:

albuming by using only media albuming hint information; and

albuming by combining media albuming hints with content-based feature values.

30. The method of claim 29, wherein in the albuming by using only media albuming hint information, a method of indexing or clustering an arbitrary j-th content m_j with an i-th label g_i by only using albuming hints is expressed as the following equation:

$L_j=g_i⨯<br><br>\mathrm{\Phi <br><br>}<br><br>\left(H_j,g_i\right),\mathrm{where}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{\Phi <br><br>}<br><br>\left(H_j,g_i\right)=\{\begin{array}{cc}1,& <br><br>\underset{l=1}{\overset{L}{\prod <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>B<br><br>\left(h_l,g_i\right)=1\\ 0,& <br><br>\mathrm{otherwise}\end{array}$

where function B(a,b) is a Boolean function in which when a=b, the function B is 1, or else 0, and the finally determined L_j is the label of a j-th content m_j.

31. The method of claim 29, wherein in the albuming by combining media albuming hints with content-based feature values, albuming hint H_j of an arbitrary j-th content m_j is combined with content-based feature value F_j, and the generated new feature value F_j is expressed as the following equation:

F_j′

=⊖

(F_j,H_j)

where ⊖

is an arbitrary function for combining a content-based feature value and an albuming hint.

32. The method of claim 31, wherein the new combined feature value is compared with a feature value learned with respect to label set G to obtain a similarity distance value, and a label having a highest similarity is determined to be a label of the j-th content m_j and, the label of the j-th content m_j is determined according to the following equation:

$L_j=\underset{g\in <br><br>G}{\arg <br><br>\text{\hspace{1em}<br><br>}<br><br>\min }<br><br>\left\{D<br><br>\left(F_j^{\mathrm{\prime <br><br>}},F_G\right)\right\}$

46. A computer readable recording medium having embodied thereon a computer program for executing the method of claim 1.

2. The method of claim 2, further comprising:

generating album metadata to manage album information of multimedia contents by using an albumed result; and

storing albumed multimedia contents and album metadata related to albuming in a database.

33. A multimedia albuming system comprising:

a media albuming hint description structure providing unit generating a media albuming hint description structure;

an albuming hint extraction unit extracting albuming hint information from multimedia contents and describing albuming hints according to the media albuming hint description structure generated by the media albuming hint description structure providing unit;

a media description unit generating a media descriptor by using the described albuming hint information; and

a media albuming unit albuming multimedia contents by using the media descriptor.

34. The system of claim 33, further comprising:

a media album description unit generating album metadata to manage album information of multimedia contents by using albumed result; and

a database storing albumed multimedia contents and album metadata related to albuming in a database.

35. The system of claim 33, further comprising:

a media acquisition unit obtaining contents from a multimedia content acquisition apparatus and performing preprocessing; and

a media input unit receiving inputs of the multimedia contents and the metadata corresponding to the multimedia contents obtained from the multimedia content acquisition apparatus.

36. The system of claim 33, wherein the albuming hint information of the albuming hint extraction unit comprises photo albuming hint information, music albuming hint information and video albuming hint information.

37. The system of claim 36, wherein the description structure of the photo albuming hint information comprises at least one of a description structure expressing information on a time when a photo is taken and camera information, a description structure expressing a perceptual characteristic of human beings with respect to a contents of the photo, a description structure expressing information on a person included in the photo, a description structure expressing information on a view of the photo, and a description structure expressing information on a popularity of the photo.

38. The system of claim 36, wherein the description structure of the music albuming hint information comprises at least one of a description structure expressing information on a time when a music file is recorded, generated or edited, a description structure expressing a part that is a highlight of a music file, a description structure expressing a level of perceptual sound quality of a music file, a description structure expressing information on a mood of music, a description structure expressing information on a situation suitable to reproduce a music file, a description structure expressing media resource information on photos or moving pictures related to a music file, and a description structure expressing popularity or preference of a music file.

39. The system of claim 36, wherein the description structure of the video albuming hint information comprises a description structure expressing information on major characters included in a video file, a description structure expressing a part that is a highlight of a video file, and a description structure expressing a popularity or preference of a video file.

40. The system of claim 33, wherein the described albuming hint information is used by a media description tool to generate a media descriptor that is metadata to describe media together with content-based feature value metadata.

41. The system of claim 33, wherein the media albuming unit comprises at least one of:

a photo data albuming unit clustering or indexing photo data by using the media descriptor;

a music data albuming unit clustering or indexing music data by using the media descriptor; and

a video data albuming unit clustering or indexing video data by using the media descriptor.

42. The system of claim 41, wherein the photo data albuming unit comprises at least one of:

a situation-based photo albuming unit albuming photos based on a situation in which a photo is taken;

a category-based photo albuming unit albuming photos based on a semantic category included in the photo; and

a person-based photo albuming unit albuming photos based on a person included in the photo.

43. The system of claim 41, wherein the music data albuming unit comprises at least one of:

an ID3-based music albuming unit albuming music based on ID3 metadata including at least one of a title of a music file, a singer'"'"'s album, a genre, a language, and reproduction time information; and

a mood-based music albuming unit albuming music based on a mood of a music file.

44. The system of claim 41, wherein the video data albuming unit comprises at least one of:

a shot-based video albuming unit albuming video data based on a basic unit shot of a video segment;

a scene-based video albuming unit albuming video data based on a scene having semantic information in addition to a shot;

a genre-based video albuming unit albuming video data based on a genre of a video file; and

a person-based video albuming unit albuming based on a person included in a video file.

45. The system of claim 33, wherein the media albuming unit performs albuming by using only media albuming hint information or by combining media albuming hints with content-based feature values.