Method and apparatus for encoding multimedia contents and method and system for applying encoded multimedia contents

• US 20070086665A1
• Filed: 07/20/2006
• Published: 04/19/2007
• Est. Priority Date: 07/20/2005
• Status: Abandoned Application

• Current Assignee: Korea Advanced Institute of Science and Technology, Samsung Electronics Co. Ltd.
• Original Assignee: Research and Industrial Cooperation Group, Samsung Electronics Co. Ltd.
• Inventors: Kim, Sangkyun, Moon, Youngsu, Kim, Jiyeun, Ro, Yongman, Yang, Seungil
• Application Number:
  US11/489,478
• Publication Number:
  US 20070086665A1
• Time in Patent Office: Days
• US Class Current: 382/239
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

A method and an apparatus for encoding multimedia contents, and a method and a system for applying multimedia contents are provided. The method for applying multimedia contents includes: storing an MAF file in a database, the MAF file including a header with location information that provides the location of media data, at least one single track with media data and metadata compatible with a predetermined standard, utilization data representing information for media application method; and browsing or sharing the MAF file stored in the database. The media metadata compatible with the predetermined standard is at least one of media player metadata and a media album metadata. Accordingly, even when the user does not have a specific application or a function for applying metadata, general-purpose multimedia content files can be effectively used by effectively browsing or sharing the multimedia content files.

Claims

1. A method of encoding multimedia contents, comprising:

separating media data and metadata from multimedia contents;

creating metadata complying with a predetermined standard format by using the separated metadata; and

encoding the media data and the metadata complying with the standard format, and thus creating a multimedia application format (MAF) file including a header containing information indicating a location of the media data, the metadata and the media data,

wherein the metadata complying with the standard format includes media player metadata.

2. The method of claim 1, further comprising obtaining the multimedia content from a multimedia apparatus or inputting the multimedia content.

3. The method of claim 2, wherein the multimedia contents comprise photos acquired from a photo content acquiring apparatus and music and video data related to the photos.

4. The method of claim 1, wherein the separating of media data and metadata from multimedia contents comprises extracting information required to generate metadata related to a corresponding media content by parsing exchangeable image file format (Exif) metadata or decoding a joint photographic experts group (JPEG) image included in the multimedia contents.

5. The method of claim 4, wherein the metadata comprises Exif metadata of a JPEG photo file, ID3 metadata of an MP3 music file, and compression related metadata of an MPEG video file.

6. The method of claim 1, wherein in the creating of the metadata complying with a predetermined standard format, the metadata complying with an MPEG standard is created from the separated metadata, or the metadata complying with an MPEG standard is created by extracting and generating metadata from the media content by using an MPEG-based standardized description tool.

7. The method of claim 1, wherein the media player metadata comprises collection level metadata for grouping a plurality of photos and item level metadata for each photo.

8. The method of claim 7, wherein the item level metadata comprises:

metadata describing creation information of corresponding metadata;

metadata describing creation information of a photo that the item level metadata defines;

metadata describing identification information of a photo that the item level metadata defines; and

metadata of contents-based visual feature information of a photo that the item level metadata defines.

9. The method of claim 7, wherein the collection level metadata comprises:

metadata describing creation information of corresponding metadata;

metadata describing creation information of a photo collection that the collection level metadata defines;

metadata describing identification information of each photo in a photo collection that the collection level metadata defines; and

metadata of a sublevel photo collection in a photo collection that the collection level metadata defines.

10. The method of claim 9, wherein the metadata of each photo collection comprises a unique identifier, and also according to a corresponding identifier, comprises metadata representing a photo collection bound by an identical event, metadata representing a photo collection bound by an identical category, and metadata representing a photo collection bound by an identical person.

11. The method of claim 9, wherein the metadata of the photo collection that the collection level metadata defines is expressed by ContentCollection DS of MPEG-7.

12. The method of claim 10, wherein the identifier of a photo collection is defined by “

Event”

representing an event collection, “

Category”

representing a category collection, and “

Person”

representing a person collection.

13. The method of claim 10, wherein the metadata of an event collection comprises metadata representing a title of a corresponding event collection, and metadata representing a photographing time of photos in the event collection.

14. The method of claim 13, wherein the metadata representing a title of an event collection is expressed in “

//CreationInformationDS/Creation/Title”

of MPEG-7 DS, and the metadata representing a photographing time of photos in the event collection is expressed in “

//CreationInformationDS/Creation/CreationCoordinates/Date”

of MPEG-7 MDS when expressing the metadata of an event collection.

15. The method of claim 10, wherein the metadata of a category collection comprises metadata representing a title of a corresponding category collection.

16. The method of claim 15, wherein the metadata representing a title of a category collection is expressed in “

//Creation Information DS/Classification/Subject/KeywordAnnotation/Keyword/”

”

of MPEG-7 DS for expressing the metadata of a category collection.

17. The method of claim 10, wherein the person collection metadata comprises metadata representing a name of a person corresponding to a person collection.

18. The method of claim 17, wherein metadata representing a name of a person of the person collection, which is “

//CreatinoInformationDS/Creation/Creator/Role”

of MPEG-7 MDS, is represented in “

Actor”

, and a name of a corresponding person is expressed using “

//CreationInformationDS/Creation/Creator/Agent.

19. The method of claim 1, wherein the metadata complying with the standard format further comprises media album metadata.

20. The method of claim 19, wherein the media album metadata comprises at least one of content-based feature values of the media content, media group information, media application information, right information on a media album, and media albuming hints.

21. The method of claim 20, wherein the content-based feature values of the media data comprises:

a visual descriptor that is metadata related to the visual feature information of the media data; and

an audio descriptor that is metadata related to the audio feature information of the media data.

22. The method of claim 20, wherein the media group information is expressed using MPEG-7 multimedia description scheme (MDS) or MPEG-21 digital item declaration (DID).

23. The method of claim 22, wherein the media group information comprises media creation information metadata, semantic information metadata, and contents configuration metadata for expressing the media group information using the MPEG-7 MDS.

24. The method of claim 20, wherein the media application information is a description of,

a procedure for a media album apparatus to display a plurality of photos on a screen based on metadata, described using an MPEG-4 scene descriptor, or

a procedure to display a plurality of photos on a screen based on metadata, described using MPEG-21 digital item processing (DIP), or

information to adaptively transform a photo album MAF file with respect to the performance of a terminal or a network, described using MPEG-21 digital item adaptation (DIA).

25. The method of claim 20, wherein right information on the media album comprises:

MPEG-21 intellectual property management and protection (IPMP) as an item to control other users'"'"' access to the photo album;

an item to control browsing of a media album MAF file;

an item to control printing of the media album MAF file; and

an item to control editing of the media album MAF file.

26. The method of claim 20, wherein the media albuming hints comprises:

a description structure for expressing camera information and photographing information when a photo is taken;

a description structure for expressing perceptional characteristics of a human being in relation to the contents of a photo;

a description structure for expressing view information of a camera;

a description structure for expressing information on persons included in a photo; and

a description structure for expressing popularity information of a photo.

27. The method of claim 26, wherein the description structure for expressing camera information and photographing information when a photo is taken comprises:

at least one of information on the photographer who takes a photo, time information on the time when a photo is taken, manufacturer information on the manufacturer of the camera with which a photo is taken, camera model information of a camera with which a photo is taken, shutter speed information of a shutter speed used when a photo is taken, color mode information of a color mode used when a photo is taken, information indicating the sensitivity of a film when a photo is taken, information indicating whether or not a flash is used when a photo is taken, information indicating the aperture number of a lens iris used when a photo is taken, information indicating the optical zoom distance used when a photo is taken, information indicating the focal length used when a photo is taken, information indicating the distance between the focused-upon subject and the camera when a photo is taken, global positioning system (GPS) information on a place where a photo is taken, information indicating the orientation of a first pixel of a photo image as the orientation of a camera when the photo is taken, information indicating sound recorded together when a photo is taken, and information indicating a thumbnail image stored for high-speed browsing in a camera after a photo is taken; and

information indicating whether or not the photo data includes Exif information as metadata.

28. The method of claim 26, wherein the description structure for expressing perceptional characteristics of a human being in relation to the contents of a photo comprises at least one of:

an item (avgColorfulness) indicating the colorfulness of the color tone expression of a photo;

an item (avgColorCoherence) indicating the color coherence of the entire color tone appearing in a photo;

an item (avgLevelOfDetail) indicating the detailedness of the contents of a photo;

an item (avgHomogenity) indicating the homogeneity of texture information of the contents of a photo;

an item (avgPowerOfEdge) indicating the robustness of edge information of the contents of a photo;

an item (avgDepthOfField) indicating the depth of the focus of a camera in relation to the contents of a photo;

an item (avgBlurrness) indicating the blurriness of a photo caused by shaking of a camera generally due to a slow shutter speed;

an item (avgGlareness) indicating the degree that the contents of a photo are affected by a very bright flash light or a very bright external light source when the photo is taken; and

an item (avgBrightness) indicating information on the brightness of an entire photo.

29. The method of claim 28, wherein the description structure indicating the colorfulness of the color tone expression of a photo is measured after normalizing the histogram heights of each RGB color value and the distribution value of the entire color values from a color histogram, or by using the distribution value of a color measured using a CIE L*u*v color space.

30. The method of claim 28, wherein the description structure indicating the color coherence of the entire color tone appearing in a photo can be measured by using a dominant color descriptor from among the MPEG-7 visual descriptors, and is measured by normalizing the histogram heights of each color value and the distribution value of the entire color values from a color histogram.

31. The method of claim 28, wherein the description structure indicating the detailedness of the contents of a photo is measured by using an entropy measured from the pixel information of the photo, or by using an isopreference curve that is an element for determining the actual complexity of a photo, or by using a relative measurement method in which compression ratios are compared when compressions are performed under identical compression conditions.

32. The method of claim 28, wherein the description structure indicating the homogeneity of texture information of the contents of a photo is measured by using the regularity, direction and scale of texture from feature values of a texture browsing descriptor among the MPEG-7 visual descriptors.

33. The method of claim 28, wherein the description structure indicating the robustness of edge information of the contents of a photo is measured by extracting edge information from a photo and normalizing the extracted edge power.

34. The method of claim 28, wherein the description structure indicating the depth of the focus of a camera in relation to the contents of a photo is measured by using the focal length and diameter of a camera lens, and an iris number.

35. The method of claim 28, wherein the description structure indicating the blurriness of a photo caused by shaking of a camera due to a slow shutter speed is measured by using the edge power of the contents of the photo.

36. The method of claim 28, wherein the description structure indicating the degree that the contents of a photo are affected by a very bright external light source is measured by using the brightness of the pixel value of the photo.

37. The method of claim 28, wherein the description structure indicating information on the brightness of an entire photo is measured by using the brightness of the pixel value of the photo.

38. The method of claim 26, wherein the description structure for expressing information on persons included in a photo comprises:

an item indicating the number of persons included in a photo;

an item indicating the position of the face of each person and the position of clothes worn by the person; and

an item indicating the relationship between persons included in a photo.

39. The method of claim 38, wherein the item indicating the position information of the face and clothes of each person included in a photo comprises an ID, the face position, and the position of clothes of the person.

40. The method of claim 38, wherein the item indicating the relationship between persons included in a photo comprises an item indicating a first person of the two person in the relationship, an item indicating the second person, and an item indicating the relationship between the two persons.

41. The method of claim 26, wherein the description structure for expressing the view information of the photo comprises:

an item indicating whether the main subject of a photo is a background or a foreground;

an item indicating the position of a part corresponding to the background of a photo in the contents expressed in the photo;

an item indicating the position of a part corresponding to the background of a photo.

42. The method of claim 1, further comprising creating MAF application method data, wherein in the encoding of the media data and the metadata complying with the standard format, and thus the creating of the MAF file, the MAF file including the header containing information indicating the media data, the metadata and the media data is created using the media data, the metadata complying with the standard format, and the MAF application method data.

43. The method of claim 42, wherein the MAF application method data comprises:

an MPEG-4 scene descriptor for the MAF application method data for describing an albuming method defined by a media albuming tool and a procedure and method for media reproduction; and

an MPEG-21 DIP descriptor for processing a digital item according to an intended format and procedure.

44. The method of claim 1 or claim 42, wherein in the encoding of the media data and the metadata complying with the standard format, and thus the creating of the MAF file, the MAF file comprises a single track MAF as a basic element, in which the single track MAF is formed with one media content and corresponding metadata, and the single track MAF comprises a header related to the track, MPEG metadata, and media data.

45. The method of claim 1, wherein in the encoding of the media data and the metadata complying with the standard format, and thus the creating of the MAF file, the MAF file comprises a multi-track MAF including one or more single track MAFs, an MAF header related to the multiple tracks and MPEG metadata for the multiple tracks.

46. The method of claim 42, wherein in the encoding of the media data and the metadata complying with the standard format, and thus the creating of the MAF file, the MAF file comprises a multi-track MAF including one or more single track MAFs, an MAF header related to the multiple tracks, MPEG metadata for the multiple tracks, and data on the application method of the MAF file.

88. A computer readable recording medium having embodied thereon a computer program for executing the method of any one of claims 1 through 46.

47. A method of applying multimedia contents comprising:

storing in a database, an MAF file, including at least one single track which includes a header containing information indicating a location of media data, media data complying with a predetermined standard format, and media metadata, and application data indicating information on an application method of the media; and

browsing or sharing the MAF file stored in the database,

wherein the media data complying with a predetermined standard format is at least one of media player metadata or media album metadata.

48. The method of claim 47, wherein the MAF file physically includes the media data, or the MAF file does not physically include the media data and instead includes a locator indicating the position of another storage apparatus in which the media data exists.

49. The method of claim 48, wherein if the MAF file does not physically include the media data, the MAF file includes a thumbnail image together with the locator.

50. The method of claim 47, wherein in the storing the MAF file in the database, the MAF file is stored according to an event MAF in which photos are grouped in units of events and photos of an identical event are stored as an integrated MAF file, a person MAF in which photos are grouped based on persons and photos including an identical person are stored as an integrated MAF file, and a category MAF in which photos are grouped based on categories and photos belonging to an identical category are stored as an integrated MAF file.

51. The method of claim 47, wherein in the browsing or sharing of the MAF file stored in the database, the browsing or sharing of the MAF file is performed according to a user'"'"'s query.

52. The method of claim 51, wherein MAF files matching the user'"'"'s query are found by parsing the metadata of each MAF file stored in the database.

53. The method of claim 47, wherein in the browsing or sharing of the MAF file stored in the database, a created media album MAF file is transmitted to other devices through a communication channel.

54. The method of claim 47, wherein in the browsing or sharing of the MAF file stored in the database, the MAF file is transmitted to a device which can recognize the MAF but does not provide full compatibility, a terminal device which fully recognizes the MAF, a mobile device which fully recognizes the MAF, or to a web album device which fully recognizes the MAF.

55. The method of claim 47, wherein in the browsing or sharing of the MAF file stored in the database, the MAF file includes a presentation tool which expresses photos systematically based on metadata.

56. The method of claim 47, wherein in the browsing or sharing of the MAF file stored in the database, if the presentation tool which expresses photos systematically based on metadata does not exist in the MAF file, photo browsing is enabled by using a presentation tool of an application that the user has.

57. The method of claim 47, wherein the browsing or sharing of the MAF file stored in the database comprises:

if the device does not perform highly enough to generate metadata,

transmitting obtained photo data to a high performance album server so that metadata is created, then, receiving the data again as a photo album MAF file and thus based on the metadata, effectively browsing photos; and

the user editing part of the received MAF file in order to generate a new MAF file and transmitting the new MAF to other users.

58. The method of claim 47, wherein the browsing or sharing of the MAF file stored in the database comprises transmitting files, which are obtained from a variety of photographing apparatuses and encoded in a photo album MAF format, to a web album apparatus through the Internet.

59. The method of claim 58, wherein when a great number of photos included in the photo album MAF are stored in the web album apparatus, one MAF file is transmitted to the web album apparatus and the web album apparatus extracts metadata from the transmitted MAF file and automatically performs categorization.

60. The method of claim 58, wherein the browsing or sharing of the MAF file stored in the database comprises:

decoding the input MAF file;

performing user authentication by using the decoded metadata;

if the user is authenticated, reading the rights of the user related to a photo album;

extracting metadata according the read right;

extracting photos by using the extracted metadata; and

uploading the extracted photos on a web site.

89. A computer readable recording medium having embodied thereon a computer program for executing the method of any one of claims 47 through 60.

61. An apparatus for encoding multimedia contents, comprising:

a pre-processing unit separating media data and metadata from multimedia contents;

a metadata creation unit creating metadata complying with a predetermined standard format by using the separated metadata; and

an encoding unit encoding the media data and the metadata complying with the standard format, and thus creating an MAF file including a header containing information indicating a location of the media data, the metadata and the media data,

wherein the metadata complying with the standard format includes media player metadata.

62. The apparatus of claim 61, further comprising a media acquisition unit obtaining and/or inputting the multimedia content from a multimedia apparatus.

63. The apparatus of claim 61, wherein the multimedia contents comprise photos acquired from a photo content acquiring apparatus and music and video data related to the photos.

64. The apparatus of claim 61, wherein the pre-processing unit extracts information required to generate metadata of a corresponding media content by parsing Exif metadata or decoding a JPEG image included in the multimedia content.

65. The apparatus of claim 61, wherein the metadata creation unit creates metadata complying with an MPEG standard from the separated metadata, or the metadata complying with an MPEG standard is created by extracting and creating metadata from the media content by using an MPEG-based standardized description tool.

66. The apparatus of claim 61, wherein the media player metadata comprises collection level metadata for grouping a plurality of photos and item level metadata for each photo.

67. The apparatus of claim 66, wherein the item level metadata comprises:

metadata describing creation information of corresponding metadata;

metadata describing creation information of a photo that the item level metadata defines;

metadata describing identification information of a photo that the item level metadata defines; and

metadata of contents-based visual feature information of a photo that the item level metadata defines.

68. The apparatus of claim 66, wherein the collection level metadata comprises:

metadata describing creation information of corresponding metadata;

metadata describing creation information of a photo collection that the collection level metadata defines;

metadata describing identification information of each photo in a photo collection that the collection level metadata defines; and

metadata of a sublevel photo collection in a photo collection that the collection level metadata defines.

69. The apparatus of claim 61, wherein the metadata complying with the standard format standard format further includes media album metadata.

70. The apparatus of claim 69, wherein the media album metadata comprises at least one of content-based feature values of the media data, media group information, media application information, right information on a media album, and media albuming hints.

71. The apparatus of claim 61, further comprising an application method data creation unit creating MAF application method data, wherein the encoding unit creates an MAF file including the header containing information indicating the media data, the metadata and the media data, by using the media data, the metadata complying with the standard format, and the MAF application method data.

72. The apparatus of claim 71, wherein the MAF application method data comprises:

an MPEG-4 scene descriptor for describing an albuming method defined by a media albuming tool and a procedure and method for media reproduction; and

an MPEG-21 DIP descriptor.

73. The apparatus of claim 61 or claim 71, wherein the MAF file comprises a single track MAF as a basic element, in which the MAF file is formed with one media content and corresponding metadata, and the single track MAF comprises a header related to the track, MPEG metadata, and media data.

74. The apparatus of claim 61 or claim 71, wherein the MAF file comprises a multi-track MAF including one or more single track MAFs, an MAF header related to the multiple tracks and MPEG metadata for the multiple tracks.

75. The apparatus of claim 71, wherein the MAF file comprises a multi-track MAF including one or more single track MAFs, an MAF header related to the multiple tracks, MPEG metadata for the multiple tracks, and data on the application method of the MAF file.

76. A system for applying multimedia contents, comprising:

a database storing an MAF file, including at least one single track which includes a header containing information indicating a location of media data, media data, and media metadata, and application data indicating information on an application method of the media; and

an application unit browsing or sharing the MAF file stored in the database,

wherein media data complying with the standard format is at lease one of media player metadata and media album metadata.

77. The system of claim 76, wherein the MAF file physically includes the media data, or the MAF file does not physically include the media data and instead includes a locator indicating the position of another storage apparatus in which the media data exists.

78. The system of claim 77, wherein if the MAF file does not physically include the media data, the MAF file includes a thumbnail image together with the locator.

79. The system of claim 76, wherein the database stores the MAF file according to an event MAF in which photos are grouped in units of events and photos of an identical event are stored as an integrated MAF file, a person MAF in which photos are grouped based on persons and photos including an identical person are stored as an integrated MAF file, and a category MAF in which photos are grouped based on categories and photos belonging to an identical category are stored as an integrated MAF file.

80. The system of claim 76, further comprising a query processing unit processing a query of a user, and browsing or sharing an MAF file stored in the database according to the user'"'"'s query.

81. The system of claim 76, wherein the application unit finds MAF files matching the user'"'"'s query by parsing the metadata of each MAF file stored in the database.

82. The system of claim 76, wherein the application unit transmits the created MAF file to other devices through a communication channel.

83. The system of claim 82, wherein the application unit transmits the MAF file, to a device which can recognize the MAF but does not provide full compatibility, a terminal device which fully recognizes the MAF, a mobile device which fully recognizes the MAF, or to a web album device which fully recognizes the MAF.

84. The system of claim 76, wherein the application unit includes a presentation tool, which expresses photos systematically based on metadata, in the MAF file.

85. The system of claim 76, wherein if the presentation tool which expresses photos systematically based on metadata does not exist in the MAF file, the application unit enables photo browsing by using a presentation tool of an application of the user.

86. The system of claim 76, wherein the application unit transmits files, which are obtained from a variety of photographing apparatuses and encoded in a photo album MAF format, to a web album apparatus through the Internet.

87. The system of claim 86, wherein when a great number of photos included in the MAF file are stored in a web album apparatus, one MAF file is transmitted to the web album apparatus and the web album apparatus extracts metadata from the transmitted MAF file and automatically performs categorization.