Hybrid audio-visual categorization system and method

• US 20100125539A1
• Filed: 11/20/2009
• Published: 05/20/2010
• Est. Priority Date: 03/30/2006
• Status: Active Grant

• Current Assignee: Sony France SA (Sony Group Corp.)
• Original Assignee: Sony France SA (Sony Group Corp.)
• Inventors: Aucouturier, Jean-Julien, Pachet, Francois, Roy, Pierre
• Granted Patent: US 8,321,414 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G10H 2240/081 Genre classification, i.e. descriptive metadata for classification or selection of musical pieces according to style analysis genre G10H2210/036

Abstract

Meta-data (tags) for an audiovisual file can be generated by prompting a user to input certain tags (meta-data) descriptive of the audiovisual file, to serve as an initial estimate of the tags, and then revising the initial estimate (notably to expand it and/or render it more precise) based on the assumption that the relationships which hold between the different tags for a set of manually-tagged training examples will also hold for the tags of the input file now being tagged.

Claims

1. A method of producing a set of tags for an input audiovisual file, the set of tags indicating values of attributes of an audiovisual work of defined type represented by said audiovisual file, the method comprising the steps of:

issuing, to a user, a prompt for manual assignment of tags to said input audiovisual file;

inputting, as an initial estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file, data representative of the tags assigned to said input audiovisual file by the user in response to said prompt;

automatically applying a set of one or more correlation functions to the attribute-value estimates of said initial estimate, to produce a set of revised estimates; and

outputting the final result of the applying step as the set of tags for said input audiovisual file;

wherein the correlation functions applied in said applying step are functions embodying the correlations holding between known attribute-values of a set of training examples, said training examples being audiovisual works of said defined type corresponding to manually-tagged audiovisual files.

2. An audiovisual-file tag production method according to claim 1, wherein said prompt-issuing step comprises issuing a prompt to said user when the user accesses an audiovisual file whose tags are missing or incomplete.

3. An audiovisual-file tag production method according to claim 1, wherein said prompt-issuing step comprises issuing a prompt to said user when the user acquires said audiovisual file.

4. An audiovisual-file tag production method according to claim 1, wherein said prompt-issuing step comprises issuing a prompt to said user when the user indicates a desire for tagging of said audiovisual file to be performed.

5. An audiovisual-file tag production method according to claim 1, wherein said prompt-issuing step employs a user interface adapted to prompt said user to input tags for an audiovisual file and said user interface is adapted to indicate to the user which kind of tags should be input.

6. An audiovisual-file tag production method according to claim 1, and comprising the step of presenting to the user, for user input selected in the group consisting of ratification and correction, the revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file;

wherein, if the user input consists of correction of the revised estimate, there is further provided a step of using the corrected estimate to calculate a further-revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file.

7. An audiovisual-file tag production method according to claim 6, and comprising the step of assigning a respective confidence measure to each attribute value of said estimates, said confidence measure being set to a maximum value for attribute values provided by the user to ensure that said attribute values provided by the user are not changed.

8. An audiovisual-file tag production method according to claim 1, wherein the correlation-function application step is applied iteratively and, after each iteration, there is provided the step of determining the best estimate so far of the value of each attribute, wherein the first iteration applies correlation functions to said initial estimate of attribute-values, and subsequent iterations apply correlation functions to the best estimates so far of said attribute-values.

9. A computer readable medium storing a computer program having a set of instructions which, when executed by a computer apparatus, cause the computer apparatus to perform a method of producing a set of tags for an input audiovisual file, the set of tags indicating values of attributes of an audiovisual work of defined type represented by said audiovisual file, the method comprising the steps of:

issuing, to a user, a prompt for manual assignment of tags to said input audiovisual file;

inputting, as an initial estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file, data representative of the tags assigned to said input audiovisual file by the user in response to said prompt;

automatically applying a set of one or more correlation functions to the attribute-value estimates of said initial estimate, to produce a set of revised estimates; and

outputting the final result of the applying step as the set of tags for said input audiovisual file;

wherein the correlation functions applied in said applying step are functions embodying the correlations holding between known attribute-values of a set of training examples, said training examples being audiovisual works of said defined type corresponding to manually-tagged audiovisual files.

12. The computer readable medium according to claim 9, wherein said prompt-issuing step comprises issuing a prompt to said user when the user accesses an audiovisual file whose tags are missing or incomplete.

13. The computer readable medium according to claim 9, wherein said prompt-issuing step comprises issuing a prompt to said user when the user acquires said audiovisual file.

14. The computer readable medium according to claim 9, wherein said prompt-issuing step comprises issuing a prompt to said user when the user indicates a desire for tagging of said audiovisual file to be performed.

15. The computer readable medium according to claim 9, wherein said prompt-issuing step employs a user interface adapted to prompt said user to input tags for an audiovisual file and said user interface is adapted to indicate to the user which kind of tags should be input.

16. The computer readable medium according to claim 9, and comprising the step of presenting to the user, for user input selected in the group consisting of ratification and correction, the revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file;

wherein, if the user input consists of correction of the revised estimate, there is further provided a step of using the corrected estimate to calculate a further-revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file.

17. A computer readable medium according to claim 16, and comprising the step of assigning a respective confidence measure to each attribute value of said estimates, said confidence measure being set to a maximum value for attribute values provided by the user to ensure that said attribute values provided by the user are not changed.

18. The computer readable medium according to claim 9, wherein the correlation-function application step is applied iteratively and, after each iteration, there is provided the step of determining the best estimate so far of the value of each attribute, wherein the first iteration applies correlation functions to said initial estimate of attribute-values, and subsequent iterations apply correlation functions to the best estimates so far of said attribute-values.

10. An audiovisual-file-tagging system configured to produce a set of tags for an input audiovisual file, the set of tags indicating values of attributes of an audiovisual work of defined type represented by said audiovisual file, the system comprising:

a prompt-issuing unit configured to issue, to a user, a prompt for manual assignment of tags to said input audiovisual file;

an initial estimate unit configured to input, as an initial estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file, data representative of the tags assigned to said input audiovisual file by the user in response to said prompt;

a correlation function application unit configured to automatically apply a set of one or more correlation functions to the attribute-value estimates of said initial estimate, to produce a set of revised estimates; and

a final result outputting unit configured to output the final result of the applying step as the set of tags for said input audiovisual file;

wherein the correlation functions applied by said correlation function application unit are functions embodying the correlations holding between known attribute-values of a set of training examples, said training examples being audiovisual works of said defined type corresponding to manually-tagged audiovisual files.

11. An audiovisual-file-tagging system according to claim 10, further comprising a memory storing attribute-value data for said set of training examples.

19. An audiovisual-file-tagging system according to claim 10, wherein said prompt-issuing unit is further configured to issue a prompt to said user when the user accesses an audiovisual file whose tags are missing or incomplete.

20. An audiovisual-file-tagging system according to claim 10, wherein said prompt-issuing unit is further configured to issue a prompt to said user when the user acquires said audiovisual file.

21. An audiovisual-file-tagging system according to claim 10, wherein said prompt-issuing unit is further configured to issue a prompt to said user when the user indicates a desire for tagging of said audiovisual file to be performed.

22. An audiovisual-file-tagging system according to claim 10, wherein said prompt-issuing unit is further configured to employ a user interface adapted to prompt said user to input tags for an audiovisual file and said user interface is adapted to indicate to the user which kind of tags should be input.

23. An audiovisual-file-tagging system according to claim 10, further comprising:

a revised estimate presenting unit configured to present to the user, for user input selected in the group consisting of ratification and correction, the revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file;

wherein, if the user input consists of correction of the revised estimate, the revised estimate presenting unit is further configured to use the corrected estimate to calculate a further-revised estimate of the values of the respective attributes of the audiovisual work represented by said audiovisual file.

24. An audiovisual-file-tagging system according to claim 23, further comprising:

a confidence measure assigning unit configured to assign a respective confidence measure to each attribute value of said estimates, said confidence measure being set to a maximum value for attribute values provided by the user to ensure that said attribute values provided by the user are not changed.

25. An audiovisual-file tag production method according to claim 10, wherein the correlation-function application unit is further configured to apply the set of correlations iteratively and, after each iteration, to determine the best estimate so far of the value of each attribute, wherein the first iteration applies correlation functions to said initial estimate of attribute-values, and subsequent iterations apply correlation functions to the best estimates so far of said attribute-values.