Hybrid audio-visual categorization system and method

• US 8,321,414 B2
• Filed: 11/20/2009
• Issued: 11/27/2012
• Est. Priority Date: 03/30/2006
• Status: Expired due to Fees

• Current Assignee: Sony France SA (Sony Group Corp.)
• Original Assignee: Sony France SA (Sony Group Corp.)
• Inventors: Aucouturier, Jean-Julien, Roy, Pierre, Pachet, Francois
• Primary Examiner(s): Chbouki, Tarek
• Application Number:
  US12/591,506
• Publication Number:
  US 20100125539A1
• Time in Patent Office: 1,103 Days
• Field of Search: None
• US Class Current: 707/736
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G10H 2240/081 Genre classification, i.e. descriptive metadata for classification or selection of musical pieces according to style

Abstract

Meta-data (tags) for an audiovisual file can be generated by prompting a user to input certain tags (meta-data) descriptive of the audiovisual file, to serve as an initial estimate of the tags, and then revising the initial estimate (notably to expand it and/or render it more precise) based on the assumption that the relationships which hold between the different tags for a set of manually-tagged training examples will also hold for the tags of the input file now being tagged.

Claims

1. A method of producing a set of tags for an input audiovisual file, the set of tags indicating values of attributes of an audiovisual work of defined type represented by said audiovisual file, the method comprising the steps of:

issuing, to a user, a prompt for manual assignment of tags to said input audiovisual file;

inputting, as an initial estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file, data representative of the tags assigned to said input audiovisual file by the user in response to said prompt;

automatically applying a set of one or more correlation functions to the attribute-value estimates of said initial estimate, to produce a set of revised estimates;

assigning a respective confidence measure to each attribute value of said revised estimates; and

outputting the final result of the applying step as the set of tags for said input audiovisual file;

wherein the correlation functions applied in said applying step are functions embodying the correlations holding between known attribute-values of a set of training examples, said training examples being audiovisual works of said defined type corresponding to manually-tagged audiovisual files, and

wherein the correlation-function application step is applied iteratively, said correlation functions are applied only to attribute-values estimates associated with a confidence measure whose value exceeds a threshold, and said confidence measure is set to a maximum value for attribute values provided by the user as the initial estimate to ensure that said attribute values provided by the user as the initial estimate are not changed.

2. The audiovisual-file tag production method according to claim 1, wherein said prompt-issuing step comprises issuing a prompt to said user when the user accesses an audiovisual file whose tags are missing or incomplete.

3. The audiovisual-file tag production method according to claim 1, wherein said prompt-issuing step comprises issuing a prompt to said user when the user acquires said audiovisual file.

4. The audiovisual-file tag production method according to claim 1, wherein said prompt-issuing step comprises issuing a prompt to said user when the user indicates a desire for tagging of said audiovisual file to be performed.

5. The audiovisual-file tag production method according to claim 1, wherein said prompt-issuing step employs a user interface adapted to prompt said user to input tags for an audiovisual file and said user interface is adapted to indicate to the user which kind of tags should be input.

6. The audiovisual-file tag production method according to claim 1, and comprising the step of presenting to the user, for user input selected in the group consisting of ratification and correction, the revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file;

wherein, if the user input consists of correction of the revised estimate, there is further provided a step of using the corrected estimate to calculate a further-revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file.

7. The audiovisual-file tag production method according to claim 1, wherein, after each iteration of the correlation-function application step, there is provided the step of determining the best estimate so far of the value of each attribute, wherein the first iteration applies correlation functions to said initial estimate of attribute-values, and subsequent iterations apply correlation functions to the best estimates so far of said attribute-values.

8. A non-transitory computer readable storage medium storing a computer program having a set of instructions which, when executed by a computer apparatus, cause the computer apparatus to perform a method of producing a set of tags for an input audiovisual file, the set of tags indicating values of attributes of an audiovisual work of defined type represented by said audiovisual file, the method comprising the steps of:

issuing, to a user, a prompt for manual assignment of tags to said input audiovisual file;

inputting, as an initial estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file, data representative of the tags assigned to said input audiovisual file by the user in response to said prompt;

automatically applying a set of one or more correlation functions to the attribute-value estimates of said initial estimate, to produce a set of revised estimates;

assigning a respective confidence measure to each attribute value of said revised estimates; and

outputting the final result of the applying step as the set of tags for said input audiovisual file;

wherein the correlation functions applied in said applying step are functions embodying the correlations holding between known attribute-values of a set of training examples, said training examples being audiovisual works of said defined type corresponding to manually-tagged audiovisual files, and

wherein the correlation-function application step is applied iteratively, said correlation functions are applied only to attribute-values estimates associated with a confidence measure whose value exceeds a threshold, and said confidence measure is set to a maximum value for attribute values provided by the user as the initial estimate to ensure that said attribute values provided by the user as the initial estimate are not changed.

9. The computer readable medium according to claim 8, wherein said prompt-issuing step comprises issuing a prompt to said user when the user accesses an audiovisual file whose tags are missing or incomplete.

10. The computer readable medium according to claim 8, wherein said prompt-issuing step comprises issuing a prompt to said user when the user acquires said audiovisual file.

11. The computer readable medium according to claim 8, wherein said prompt-issuing step comprises issuing a prompt to said user when the user indicates a desire for tagging of said audiovisual file to be performed.

12. The computer readable medium according to claim 8, wherein said prompt-issuing step employs a user interface adapted to prompt said user to input tags for an audiovisual file and said user interface is adapted to indicate to the user which kind of tags should be input.

13. The computer readable medium according to claim 8, and comprising the step of presenting to the user, for user input selected in the group consisting of ratification and correction, the revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file;

wherein, if the user input consists of correction of the revised estimate, there is further provided a step of using the corrected estimate to calculate a further-revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file.

14. The computer readable medium according to claim 8, wherein, after each iteration of the correlation-function application step, there is provided the step of determining the best estimate so far of the value of each attribute, wherein the first iteration applies correlation functions to said initial estimate of attribute-values, and subsequent iterations apply correlation functions to the best estimates so far of said attribute-values.

17. The audiovisual-file-tagging system according to claim 10, wherein said prompt-issuing unit is configured to issue a prompt to said user when the user accesses an audiovisual file whose tags are missing or incomplete.

15. An audiovisual-file-tagging system configured to produce a set of tags for an input audiovisual file, the set of tags indicating values of attributes of an audiovisual work of defined type represented by said audiovisual file, the system comprising:

a prompt-issuing unit configured to issue, to a user, a prompt for manual assignment of tags to said input audiovisual file;

an initial estimate unit configured to input, as an initial estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file, data representative of the tags assigned to said input audiovisual file by the user in response to said prompt;

a correlation function application unit having a processor configured to automatically apply a set of one or more correlation functions to the attribute-value estimates of said initial estimate, to produce a set of revised estimates, said processor being further configured to assign a respective confidence measure to each attribute value of said revised estimates; and

a final result outputting unit configured to output the final result of the applying step as the set of tags for said input audiovisual file;

wherein the correlation functions applied by said correlation function application unit are functions embodying the correlations holding between known attribute-values of a set of training examples, said training examples being audiovisual works of said defined type corresponding to manually-tagged audiovisual files, and

wherein said processor is further configured to apply the correlation function application step iteratively, to apply said correlation functions only to attribute-values estimates associated with a confidence measure whose value exceeds a threshold, and to set said confidence measure to a maximum value for attribute values provided by the user as the initial estimate to ensure that said attribute values provided by the user as the initial estimate are not changed.

16. The audiovisual-file-tagging system according to claim 15, further comprising a memory storing attribute-value data for said set of training examples.

18. The audiovisual-file-tagging system according to claim 15, wherein said prompt-issuing unit is configured to issue a prompt to said user when the user acquires said audiovisual file.

19. The audiovisual-file-tagging system according to claim 15, wherein said prompt-issuing unit is configured to issue a prompt to said user when the user indicates a desire for tagging of said audiovisual file to be performed.

20. The audiovisual-file-tagging system according to claim 15, wherein said prompt-issuing unit is further configured to employ a user interface adapted to prompt said user to input tags for an audiovisual file and said user interface is adapted to indicate to the user which kind of tags should be input.

21. The audiovisual-file-tagging system according to claim 15, further comprising:

a revised estimate presenting unit configured to present to the user, for user input selected in the group consisting of ratification and correction, the revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file;

wherein, if the user input consists of correction of the revised estimate, the revised estimate presenting unit is further configured to use the corrected estimate to calculate a further-revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file.

22. The audiovisual-file tag production method according to claim 15, wherein the correlation-function application unit is further configured, after each iteration, to determine the best estimate so far of the value of each attribute, wherein the first iteration applies correlation functions to said initial estimate of attribute-values, and subsequent iterations apply correlation functions to the best estimates so far of said attribute-values.