Method and apparatus for encoding/playing multimedia contents

• US 20080018503A1
• Filed: 07/20/2006
• Published: 01/24/2008
• Est. Priority Date: 07/20/2005
• Status: Abandoned Application

• Current Assignee: Korea Advanced Institute of Science and Technology, Samsung Electronics Co. Ltd.
• Original Assignee: Research and Industrial Cooperation Group, Samsung Electronics Co. Ltd.
• Inventors: Kim, Sangkyun, Kim, Jiyeun, Yang, Seungji, Ro, Yongman
• Application Number:
  US11/489,452
• Publication Number:
  US 20080018503A1
• Time in Patent Office: Days
• US Class Current: 341/50
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/2353 specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/85406 involving a specific file format, e.g. MP4 format

Abstract

A method and an apparatus for encoding and playing multimedia contents are provided. The method includes: separating media data and metadata from the multimedia contents; creating multimedia application format (MAF) metadata by using the separated metadata, the format of the MAF metadata being predetermined; and encoding the media data and the MAF metadata to generate an MAF file including a header, the MAF metadata, and the media data, the header having information that provides a location of the media data. Accordingly, in a process of integrating digital photos and other multimedia content files into one file in the application file format MAF, visual feature information obtained from photo data and the contents of the photo images, and a variety of hint feature information for effective indexing of photos are included as metadata and content application method tools based on the metadata are included. As a result, even when the user does not have a specific application or a function for applying metadata, general-purpose multimedia content files can be effectively used by effectively browsing the multimedia content files.

Claims

1. A method of encoding multimedia contents, comprising:

separating media data and metadata from multimedia contents;

creating metadata complying with a predetermined multimedia application format (MAF) by using the separated metadata; and

encoding the media data and the metadata complying with the standard format, and thus creating an MAF file including a header containing information indicating a location of the media data, the metadata and the media data.

2. The method of claim 1, further comprising obtaining multimedia data from a multimedia apparatus before the separating of the media data and the metadata from the multimedia contents.

3. The method of claim 2, wherein the multimedia contents comprise photos acquired from a photo content acquiring apparatus and music and video data related to the photos.

4. The method of claim 1, wherein the separating of the media data and the metadata from multimedia contents comprises extracting information required to generate metadata related to a corresponding media content by parsing exchangeable image file format (Exif) metadata or decoding a joint photographic experts group (JPEG) image included in the multimedia contents.

5. The method of claim 4, wherein the metadata comprises Exif metadata of a JPEG photo file, ID3 metadata of an MP3 music file, and compression related metadata of an MPEG video file.

6. The method of claim 1, wherein in the creating of the metadata complying with a predetermined standard format, the metadata complying with an MPEG standard is created from the separated metadata, or the metadata complying with an MPEG standard is created by extracting and creating metadata from the media content by using an MPEG-based standardized description tool.

7. The method of claim 6, wherein the metadata complying with an MPEG standard comprises MPEG-7 metadata for a media content itself, and MPEG-21 metadata for declaration, adaptation conversion, and distribution of the media content.

8. The method of claim 7, wherein the MPEG-7 metadata comprises MPEG-7 descriptors of metadata for media content-based feature values, MPEG-7 semantic descriptors of metadata for media semantic information, and MPEG-7 media information/creation descriptors of media creation information.

9. The method of claim 8, wherein the MPEG-7 media information/creation descriptors comprise media albuming hints.

10. The method of claim 9, wherein the media albuming hints comprises acquisition hints expressing camera information and photographing information when a photo is taken, perception hints expressing perceptional characteristics of a human being in relation to the contents of a photo, view hints expressing view information of a camera, subject hints expressing information on persons included in a photo, and popularity expressing popularity information of a photo.

11. The method of claim 10, wherein the acquisition hints expressing camera information and photographing information when a photo is taken comprises:

at least one of information on the photographer who takes a photo, time information on the time when a photo is taken, manufacturer information on the manufacturer of the camera with which a photo is taken, camera model information of a camera with which a photo is taken, shutter speed information of a shutter speed used when a photo is taken, color mode information of a color mode used when a photo is taken, information indicating the sensitivity of a film when a photo is taken, information indicating whether or not a flash is used when a photo is taken, information indicating the aperture number of a lens iris used when a photo is taken, information indicating the optical zoom distance used when a photo is taken, information indicating the focal length used when a photo is taken, information indicating the distance between the focused-upon subject and the camera when a photo is taken, global positioning system (GPS) information on a place where a photo is taken, information indicating the orientation of a first pixel of a photo image as the orientation of a camera when the photo is taken, information indicating sound recorded together when a photo is taken, and information indicating a thumbnail image stored for high-speed browsing in a camera after a photo is taken; and

information indicating whether or not the photo data includes Exif information as metadata.

12. The method of claim 10, wherein the perception hints expressing perceptional characteristics of a human being in relation to the contents of a photo comprises at least one of:

an item (avgcolorfulness) indicating the colorfulness of the color tone expression of a photo;

an item (avgColorCoherence) indicating the color coherence of the entire color tone appearing in a photo;

an item (avgLevelOfDetail) indicating the detailedness of the contents of a photo;

an item (avgHomogenity) indicating the homogeneity of texture information of the contents of a photo;

an item (avgPowerOfEdge) indicating the robustness of edge information of the contents of a photo;

an item (avgDepthOfField) indicating the depth of the focus of a camera in relation to the contents of a photo;

an item (avgBlurrness) indicating the blurriness of a photo caused by shaking of a camera generally due to a slow shutter speed;

an item (avgGlareness) indicating the degree that the contents of a photo are affected by a very bright flash light or a very bright external light source when the photo is taken; and

an item (avgBrightness) indicating information on the brightness of an entire photo.

13. The method of claim 12, wherein the avgcolorfulness item indicating the colorfulness of the color tone expression of a photo is measured after normalizing the histogram heights of each RGB color value and the distribution value of the entire color values from a color histogram, or by using the distribution value of a color measured using a CIE L*u*v color space.

14. The method of claim 12, wherein the avgColorCoherence item indicating the color coherence of the entire color tone appearing in a photo can be measured by using a dominant color descriptor from among the MPEG-7 visual descriptors, and is measured by normalizing the histogram heights of each color value and the distribution value of the entire color values from a color histogram.

15. The method of claim 12, wherein the avgLevelOfDetail item indicating the detailedness of the contents of a photo is measured by using an entropy measured from the pixel information of the photo, or by using an isopreference curve that is an element for determining the actual complexity of a photo, or by using a relative measurement method in which compression ratios are compared when compressions are performed under identical compression conditions.

16. The method of claim 12, wherein the avgHomogenity item indicating the homogeneity of texture information of the contents of a photo is measured by using the regularity, direction and scale of texture from feature values of a texture browsing descriptor among the MPEG-7 visual descriptors.

17. The method of claim 12, wherein the avgPowerOfEdge item indicating the robustness of edge information of the contents of a photo is measured by extracting edge information from a photo and normalizing the extracted edge power.

18. The method of claim 12, wherein the avgDepthOfField item indicating the depth of the focus of a camera in relation to the contents of a photo is measured by using the focal length and diameter of a camera lens, and an iris number.

19. The method of claim 12, wherein the avgBlurrness item indicating the blurriness of a photo caused by shaking of a camera due to a slow shutter speed is measured by using the edge power of the contents of the photo.

20. The method of claim 12, wherein the avgGlareness item indicating the degree that the contents of a photo are affected by a very bright external light source is measured by using the brightness of the pixel value of the photo.

21. The method of claim 12, wherein the avgBrightness item indicating information on the brightness of an entire photo is measured by using the brightness of the pixel value of the photo.

22. The method of claim 10, wherein the subject hints expressing information on persons included in a photo comprises:

an item indicating the number of persons included in a photo;

an item indicating the position of the face of each person and the position of clothes worn by the person; and

an item indicating the relationship between persons included in a photo.

23. The method of claim 22, wherein the item indicating the position information of the face and clothes of each person included in a photo comprises an ID, the face position, and the position of clothes of the person.

24. The method of claim 22, wherein the item indicating the relationship between persons included in a photo comprises an item indicating a first person of the two person in the relationship, an item indicating the second person, and an item indicating the relationship between the two persons.

25. The method of claim 10, wherein the view hints expressing the view information of the photo comprises:

an item indicating whether the main subject of a photo is a background or a foreground;

an item indicating the position of a part corresponding to the background of a photo in the contents expressed in the photo;

an item indicating the position of a part corresponding to the background of a photo.

26. The method of claim 7, wherein the MPEG-21 metadata comprises an MPEG-21 DID (digital item declaration) description that is metadata related to a DID, an MPEG-21 DIA (digital item adaptation) description that is metadata for a DIA, and rights expression data that is metadata regarding rights/copyrights of contents.

27. The method of claim 26, wherein the rights expression data comprises a browsing permission that is metadata of permission information of browsing photo contents, and an editing permission that is metadata of permission information of editing photo contents.

28. The method of claim 1, further comprising creating MAF application method data, wherein in the encoding of the media data and the metadata complying with the standard format, and thus the creating of the MAF file, the MAF file including the header containing information indicating the media data, the metadata and the media data is created using the media data, the metadata complying with the standard format, and the MAF application method data.

29. The method of claim 28, wherein the MAF application method data comprises:

an MPEG-4 scene descriptor for the MAF application method data for describing an albuming method defined by a media albuming tool and a procedure and method for media reproduction; and

an MPEG-21 DIP descriptor for processing a digital item according to an intended format and procedure.

30. The method of claim 1 or 29, wherein in the encoding of the media data and the metadata complying with the standard format, and thus the creating of the MAF file, the MAF file comprises a single track MAF as a basic element, in which the single track MAF is formed with one media content and corresponding metadata, and the single track MAF comprises a header related to the track, MPEG metadata, and media data.

31. The method of claim 1, wherein in the encoding of the media data and the metadata complying with the standard format, and thus the creating of the MAF file, the MAF file comprises a multi-track MAF including one or more single track MAFs, an MAF header related to the multiple tracks and MPEG metadata for the multiple tracks.

32. The method of claim 30, wherein in the encoding of the media data and the metadata complying with the standard format, and thus the creating of the MAF file, the MAF file comprises a multi-track MAF including one or more single track MAFs, an MAF header related to the multiple tracks, MPEG metadata for the multiple tracks, and data on the application method of the MAF file.

33. The method of claim 8, wherein the MPEG-7 semantic descriptors extract and generate semantic information of the multimedia contents using albuming hints.

34. The method of claim 33, wherein the extracting of the semantic information comprises performing media albuming by using media albuming hints or using the media albuming hints and the contents-based feature values.

35. The method of claim 3, wherein the performing of media albuming by using media albuming hints comprises performing indexing or clustering of an arbitrary j-th content m_j using albuming hints, which is expressed in the following Equation, where a boolean function B(a,b)=1 when a=b, and otherwise 0, and L_j represents a j-th content m_j label.

36. The method of claim 35, wherein the performing of the media albuming by using a combination of the media albuming hints and the contents-based feature values comprises combining albuming hints H_j of an arbitrary j-th content m_j and contents-based feature values F_j to create a combined new feature values, the combined new feature values F_j′

being expressed in the following Equation, where represents an arbitrary function combining the contents-based feature values and the albuming hints.

37. The method of claim 35, wherein the performing of the media albuming by using a combination of the media albuming hints and the contents-based feature values comprises obtaining a similarity value by comparing an feature value that is learned from an albuming label set G, and determining a label with the largest similarity value as a label of a j-th content m_j, the determining of the label of the j-th content m_j expressed in the following Equation.

46. The apparatus of claim 7, wherein the MPEG-7 metadata comprises MPEG-7 descriptors of metadata for media content-based feature values, MPEG-7 semantic descriptors of metadata for media semantic information, and MPEG-7 media information/creation descriptors of media creation information.

47. The apparatus of claim 46, wherein the MPEG-7 media information/creation descriptors comprise media albuming hints.

48. The apparatus of claim 46, wherein the MPEG-21 metadata comprises an MPEG-21 DID description that is metadata related to a DID, an MPEG-21 DIA description that is metadata for a DIA, and rights expression data that is metadata regarding rights/copyrights of contents.

59. A computer-readable recording medium comprising a computer-readable program for executing the method of any one of claims 1 to 39.

38. A method of playing multimedia contents, comprising:

decoding an MAF file including a header and application data to extract media data, media metadata, and application data, the header having information that provides a location of media data, the application data providing media application method information having at least one single track with media data and media metadata; and

playing the multimedia contents using the extracted metadata and the application data.

39. The method of claim 38, wherein the playing of the multimedia contents comprises using media metadata tools for processing media metadata and application method tools for browsing the media contents through metadata and application data.

40. An apparatus for encoding multimedia contents, comprising:

a pre-processing unit separating media data and metadata from multimedia contents;

a media metadata creation unit creating MAF metadata by using the separated metadata, the format of the MAF metadata being predetermined; and

an encoding unit encoding the media data and the MAF metadata to generate an MAF file including a header, the MAF metadata, and the media data, the header having information that provides a location of the media data.

41. The apparatus of claim 40, further comprising a media acquisition/input unit acquiring multimedia contents from a multimedia device or having multimedia contents input from a multimedia device.

42. The apparatus of claim 41, wherein the multimedia contents comprise photos acquired from a photo content acquiring apparatus and music and video data related to the photos.

43. The apparatus of claim 40, wherein the pre-processing unit extracts information required to generate metadata related to a corresponding media content by parsing exchangeable image file format (Exif) metadata or decoding a joint photographic experts group (JPEG) image included in the multimedia contents.

44. The apparatus of claim 40, wherein the media metadata creation unit creates the metadata complying with an MPEG standard from the separated metadata, or the metadata complying with an MPEG standard by extracting and creating metadata from the media content by using an MPEG-based standardized description tool.

45. The apparatus of claim 44, wherein the metadata compatible with the MPEG standard comprises MPEG-7 metadata for a media content, and MPEG-21 metadata for declaration, adaptation conversion, and distribution of the media content.

49. The apparatus of claim 40, further comprising an application method data creation unit that creates MAF application method data, wherein the encoding unit creates an MAF file including a header, metadata, and media data using the media data, the MAF metadata, and the MAF application method data, the header having information that provides a location of the media data.

50. The apparatus of claim 49, wherein the MAF application method data comprises:

an MPEG-4 scene description describing an albuming method defined by a media albuming tool, and a procedure and a method for media playing; and

an MPEG-21 digital item processing (DIP) descriptor for DIP according to an intended format and procedure.

51. The apparatus of claim 40 or 50, wherein the MAF file comprises a single track MAF as a basic element, in which the single track MAF is formed with one media content and corresponding metadata, and the single track MAF comprises a header related to the track, MPEG metadata, and media data.

52. The apparatus of claim 40, wherein the MAF file comprises a multi-track MAF including one or more single track MAFs, an MAF header related to the multiple tracks and MPEG metadata for the multiple tracks.

53. The apparatus of claim 51, wherein the MAF file comprises a multi-track MAF including one or more single track MAFs, an MAF header related to the multiple tracks, MPEG metadata for the multiple tracks, and data on the application method of the MAF file.

54. An apparatus for playing multimedia contents, comprising:

an MAF decoding unit decoding an MAF file including a header having information that provides a location of media data, at least one single track having media data and media metadata, and application data representing media application method information to extract the media data, media metadata, and the application data; and

an MAF playing unit playing the multimedia contents by using the extracted metadata and application data.

55. The apparatus of claim 54, further comprising media metadata tools for processing media metadata and application method tools for browsing the multimedia contents by using the metadata and the application data.

56. An MAF comprising a single track MAF corresponding to one media content, the single track MAF including an MAF header for a corresponding track, MPEG metadata, and media data.

57. An MAF comprising a multi-track MAF including one or more single track MAFs, an MAF header related to the multiple tracks, and MPEG metadata for the multiple tracks.

58. The MAF of claim 57, further comprising application method data for an application method of an MAF file.