Device and method for modality conversion of multimedia contents

• US 7,853,864 B2
• Filed: 02/13/2004
• Issued: 12/14/2010
• Est. Priority Date: 02/13/2003
• Status: Expired due to Fees

• Current Assignee: Electronics and Telecommunications Research Institute, Korea Advanced Institute of Science and Technology
• Original Assignee: Electronics and Telecommunications Research Institute, Korea Advanced Institute of Science and Technology
• Inventors: Kim, Jin-Woong, Jung, Yong-Ju, Hong, Jin-Woo, Ro, Yong-Man, Nam, Je-Ho
• Primary Examiner(s): Hutton; Doug
• Assistant Examiner(s): Debrow; James J
• Application Number:
  US10/545,632
• Publication Number:
  US 20060294546A1
• Time in Patent Office: 2,496 Days
• Field of Search: 715/201, 715/239
• US Class Current: 715/201
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 21/234336 by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/25833 involving client hardware characteristics, e.g. manufacturer, processing or storage capabilities allocation of resources considering hardware capabilities in multiprogramming arrangements G06F9/5044; allocation of resources considering software capabilities in multiprogramming arrangements G06F9/5055
• CPC Class Codes: H04N 21/26216 involving the channel capacity, e.g. network bandwidth admission control, resource allocation in open networks H04L12/5692; flow control in packet networks H04L47/10; establishing a schedule or organising the servicing of application requests taking into account QoS H04L67/61
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 7/17336 Handling of requests in head-ends

Abstract

The present invention relates to a method of systematically and synthetically accessing modality conversion that is an important part in the contents adaptive conversion process of a universal multimedia access system. The present invention provides an effective method of solving a problem, which is incurred at the time of modality conversion and still remains as one of difficult problems incurred during adaptive contents conversion. For this purpose, the present invention includes overlapped contents modeling newly proposed to determine modality conversion, a method of flexibly and clearly expressing and applying user preference for the modality conversion, and a resource allocation method of distributing resources among complicated contents based on the user preference. As a result, the integration of the above three methods provides a synthetic solution, particularly, to a problem incurred in the modality conversion and, generally, to a problem incurred in the adaptive conversion of contents.

Claims

1. A device for adaptively converting a modality of multimedia contents, the device comprising:

an input for receiving data specifying multimedia contents, metadata of the multimedia contents, and user preference for the modality conversion; and

a converter for converting the modality of the input multimedia contents;

wherein the input is adapted for receiving a multimedia contents item which comprises a unit expressible in any one of different modalities, the unit being the whole of the multimedia contents item or being a portion of the multimedia contents item, wherein the user preference for the modality conversion comprises a qualitative preference and a quantitative preference, wherein the qualitative preference indicates a sequence of modalities according to the user preference for the modalities, and the quantitative preference indicates a relative quantitative importance of at least two of the modalities to the user;

wherein the converter is for expressing the unit in a modality in which the unit is to be transmitted to the user, the converter using at least the qualitative preference in determining the modality in which the unit is to be transmitted;

wherein the converter comprises;

means for predicting an overlapped contents model using the input multimedia contents and input metadata of the contents, the overlapped contents model specifying, for each said modality and an associated range of resource values, a contents value curve which specifies, for the associated modality, a relationship between (i) a resource value in the associated range of resource values and (ii) a content value, wherein at least two of the contents value curves intersect inside one of the ranges;

means for modifying the overlapped contents model using the user preference for the modality conversion; and

means for determining a resource restriction and expressing the multimedia contents according to the resource restriction in a form corresponding to a maximum contents value of the modified overlapped contents model.

2. The device according to claim 1, wherein the means for modifying the overlapped contents model is for additionally modifying the overlapped contents model depending on a type of a terminal to which the multimedia contents are to be transmitted.

3. The device according to claim 1, wherein:

the means for modifying is to remove or not remove one or more of the contents value curves from the overlapped contents model in accordance with the qualitative preference; and

wherein the means for modifying is for adjusting the contents value curves in accordance with the quantitative preference.

4. The device according to claim 1, wherein the user preference for the modality conversion is described in an eXtended Markup Language (XML) form.

5. The device according to claim 1, wherein the user preference for the modality conversion comprises user preference for general contents modality conversion and user preference for specific modality conversion.

6. The device according to claim 1, wherein the user preference for the modality conversion comprises preference for video modality conversion, preference for image modality conversion, preference for audio modality conversion, and preference for text modality conversion.

7. The device of claim 1 wherein the unit is associated with video modality, and unit is expressible in the video modality and at least one of image modality, audio modality, and text modality.

8. The device of claim 1 wherein the quantitative preference comprises a weight assigned to each modality and indicating the modality'"'"'s relative quantitative importance to the user.

9. A method of adaptively converting a modality of multimedia contents, the method comprising:

receiving, by a device, multimedia contents and metadata of the multimedia contents from an outside source, wherein the multimedia contents comprise a multimedia contents item comprising a unit expressible in any one of different modalities, the unit being the whole of the multimedia contents item or being a portion of the multimedia contents item;

receiving, by the device, user preference for modality conversion, wherein the user preference for the modality conversion comprises a qualitative preference and a quantitative preference, wherein the qualitative preference indicates a sequence of modalities according to the user preference for the modalities, and the quantitative preference indicates a relative quantitative importance of at least two of the modalities to the user;

adaptively converting, by the device, a modality of the input multimedia contents according to the metadata of the contents and at least the quantitative preference; and

outputting, by the device, the converted multimedia contents;

wherein the method further comprises additionally receiving a resource restriction, wherein the adaptive conversion of the modality is performed while additionally considering the resource restriction;

wherein the adaptive converting of the modality comprises;

predicting an overlapped contents model using the input multimedia contents and the input metadata of the contents, the overlapped contents model specifying, for each said modality and an associated range of resource values, a contents value curve which specifies, for the associated modality, a relationship between (i) a resource value in the associated range of resource values and (ii) a content value, wherein at least two of the contents value curves intersect inside one of the ranges;

modifying the overlapped contents model using the user preference for modality conversion; and

converting a modality of the multimedia contents to allow the multimedia contents to be represented in a form corresponding to a maximum contents value of the modified overlapped contents model under the resource restriction.

10. The method according to claim 9, wherein the quantitative preference is represented by a weight of each modality, wherein the weights indicate relative quantitative importance of each modality to the user.

11. The method according to claim 9, wherein the user preference for modality conversion is described in an XML form.

12. The method according to claim 9, wherein the user preference for modality conversion comprises user preference for general contents modality conversion and user preference for specific modality conversion.

13. The adaptive conversion method according to claim 9, wherein the user preference for the modality conversion comprises preference for expressing the multimedia contents item in video modality, in image modality, in audio modality, and in text modality.

14. The method according to claim 13, wherein the multimedia contents item is received in video modality.

15. The adaptive conversion method according to claim 9, wherein the multimedia contents item is received in image modality, and the user preference for modality conversion comprises the user preference for expressing the multimedia contents item in image modality, in audio modality, and in text modality.

16. The method according to claim 9, wherein the multimedia contents item is received in audio modality, and the user preference for modality conversion comprises the user preference for expressing the multimedia contents item in audio modality and in text modality.

17. A device for adaptively converting a modality of multimedia contents, the device comprising:

means for receiving a multimedia contents item and metadata of the multimedia contents item from an outside source;

means for receiving user preference for modality conversion of the multimedia item, the user preference being represented by a sequence of modalities and a weight of each modality, wherein;

the sequence indicates a sequence of modalities according to the user preference for the modalities; and

the weights indicate relative quantitative importance of each modality to the user;

means for obtaining an overlapped contents model specifying, for each said modality and an associated range of resource values, a contents value curve which specifies, for the associated modality, a relationship between (i) a resource value in the associated range of resource values and (ii) a content value, wherein at least two of the contents value curves intersect inside one of the ranges;

means for modifying the overlapped contents model using the user preference for the modality conversion;

means for adaptively converting a modality of the input multimedia contents item according to the metadata of the contents and the user preference for the modality conversion; and

means for outputting the converted multimedia contents item;

wherein the means for modifying is to remove or not remove one or more of the contents value curves from the overlapped contents model in accordance with the sequence;

wherein the means for modifying is for adjusting the contents value curves in accordance with the weights if the weights are not equal to each other; and

the means for adaptively converting is for converting based on the overlapped contents model generated by the means for modifying.

18. A method for providing a multimedia contents item to a user'"'"'s terminal over a network, the method comprising:

(1) obtaining, by a device, data specifying;

(i) a unit, wherein the unit is a portion or a whole of the multimedia contents item and the unit is expressed in at least a first modality, wherein the unit is also expressible in one or more modalities other than the first modality, (ii) user preference for modality of the multimedia contents item, and (iii) metadata providing an indication of preferences in conversion of the multimedia contents item from the first modality to the one or more modalities other the first modality;

(2) for the unit performing operations of;

(2A) obtaining, by the device, a resource restriction, and determining from the resource restriction, the metadata and the user preference whether or not the unit is to be converted to a modality other than the first modality before transmission to the terminal;

(2B) if it is determined by the device, in operation (2A), that the unit is to be modality-converted, then performing, by the device, modality conversion of the unit and transmitting the modality-converted unit over the network to the terminal, wherein modality conversion is conversion to a different modality;

(2C) if it is determined by the device, in operation (2A), that the unit is NOT to be modality-converted, then transmitting the unit over the network to the terminal without modality conversion of the unit;

wherein the metadata comprise data representing a contents model which associates resources with content values for the modalities; and

the method further comprises;

(3) determining, by the device, whether or not the contents model is to be modified based on the user preference;

(4) if the user preference indicates that the contents model is to be modified, then modifying, by the device, the contents model based on the user preference;

(5) if the user preference indicates that the contents model is NOT to be modified, then NOT modifying, by the device, the contents model based on the user preference;

wherein operation (2A) is performed using the contents model provided by operations (3)-(5);

wherein the operation (2A) maximizes the contents value based on the contents model provided by the operations (3)-(5);

wherein the contents model in the metadata is an overlapped contents model which specifies, for each said modality and an associated range of resource values, a contents value curve which specifies, for the associated modality, a relationship between (i) a resource value in the associated range of resource values and (ii) a content value, wherein at least two of the contents value curves intersect inside one of the ranges;

wherein the user preference comprises a weight associated with each of one or more of the modalities, and at least two of the modalities are associated with different weights; and

in the operation (4), the associated contents value curves are modified to increase the content values associated with the modality of a greater weight relative to the content values associated with the modality of a smaller weight.

19. The method of claim 18 wherein the user preference indicates a sequence of modalities according to the user preference for the modalities;

and

if said sequence indicates a higher user preference for a first modality over a second modality, but the second modality'"'"'s contents value curve indicates a higher content value than the first modality'"'"'s contents value curve at least for all the resource values higher than a predefined resource value, then operation (4) is performed in which the content curve associated with the second modality is removed from the overlapped contents model.

20. A device for providing a multimedia contents item to a terminal over a network, the device being for:

(1) obtaining data specifying;

(i) a unit, wherein the unit is a portion or a whole of the multimedia contents item, and the unit is expressed in at least a first modality, wherein the unit is also expressible in one or more modalities other than the first modality, and (ii) user preference for modality of the multimedia contents item;

(2) for the unit, performing operations of;

(2A) obtaining, by the device, a resource restriction, and determining from the resource restriction and the user preference whether or not the unit is to be converted to a modality other than the first modality before transmission to the terminal;

(2B) if it is determined by the device, in operation (2A), that the unit is to be modality-converted, then performing, by the device, modality conversion of the unit and transmitting the modality-converted unit over the network to the terminal, wherein modality conversion is conversion to a different modality;

(2C) if it is determined by the device, in operation (2A), that the unit is NOT to be modality-converted, then transmitting the unit over the network to the terminal without modality conversion of the unit;

wherein operation (1) also comprises receiving metadata providing an indication of preferences in conversion of the multimedia contents item from the first modality to the one or more modalities other the first modality; and

wherein the determining operation in (2A) depends on the metadata, and the metadata comprise data representing a contents model which associates resources with content values for the modalities; and

the method further comprises;

(3) determining, by the device, whether or not the contents model is to be modified based on the user preference;

(4) if the user preference indicates that the contents model is to be modified, then modifying, by the device, the contents model based on the user preference;

(5) if the user preference indicates that the contents model is NOT to be modified, then NOT modifying, by the device, the contents model based on the user preference;

wherein operation (2A) is performed using the contents model provided by operations (3)-(5);

wherein the operation (2A) maximizes the contents value based on the contents model provided by the operations (3)-(5);

wherein the contents model in the metadata is an overlapped contents model which specifies, for each said modality and an associated range of resource values, a contents value curve which specifies, for the associated modality, a relationship between (i) a resource value in the associated range of resource values and (ii) a content value, wherein at least two of the contents value curves intersect inside one of the ranges;

wherein the user preference comprises a weight associated with each of one or more of the modalities; and

if at least two of the modalities are associated with different weights, then operation (4) is performed in which the associated contents value curves are modified to increase the content values associated with the modality of a greater weight relative to the content values associated with the modality of a smaller weight.

21. The device of claim 20 wherein the user preference indicates a sequence of modalities, the sequence indicating an order of user preference for the modalities;

and

if said sequence indicates a higher user preference for a first modality over a second modality, but the second modality'"'"'s contents value curve indicates a higher content value than the first modality'"'"'s contents value curve at least for all the resource values higher than a predefined resource value, then operation (4) is performed in which the content curve associated with the second modality is removed from the overlapped contents model.