Hybrid audio-visual categorization system and method

• US 8,392,414 B2
• Filed: 03/23/2007
• Issued: 03/05/2013
• Est. Priority Date: 03/30/2006
• Status: Expired due to Fees

• Current Assignee: Sony France SA (Sony Group Corp.)
• Original Assignee: Sony France SA (Sony Group Corp.)
• Inventors: Aucouturier, Jean-Julien, Roy, Pierre, Pachet, Francois
• Primary Examiner(s): Chbouki, Tarek
• Application Number:
  US11/690,553
• Publication Number:
  US 20080040362A1
• Time in Patent Office: 2,174 Days
• Field of Search: 707/1, 707/736
• US Class Current: 707/736
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G10H 2240/081 Genre classification, i.e. descriptive metadata for classification or selection of musical pieces according to style

Abstract

Meta-data (tags) for an audiovisual file can be generated by producing an initial estimate of the tags and then revising the estimate (notably to expand it and/or render it more precise) based on the assumption that the relationships which hold between the different tags for a set of manually-tagged training examples will also hold for the tags of the input file now being tagged. A fully-automatic method and system is a hybrid between signal-based and machine-learning approaches, because the initial tag estimate is based on the physical properties of the signal representing the audiovisual file. The initial tag estimate may be produced by inferring that the input content will have the same tags as those files of the same kind, in the training database, which have a global similarity to the input audiovisual file in terms of signal properties.

Claims

1. A fully-automatic method of producing a set of tags for an input audiovisual file, the set of tags indicating values of a plurality of attributes of an audiovisual work of a defined type represented by said audiovisual file, the method comprising:

analyzing properties of the audiovisual work represented by said input audiovisual file and evaluating a set of one or more features characterizing said properties of said audiovisual work;

providing an initial estimate of said set of tags by automatically converting said set of features evaluated in the analyzing step to said initial estimate based on first correlations between physical properties of audiovisual works of said defined type and tags applicable to audiovisual works of said defined type;

automatically applying, to the tags of said initial estimate, a set of one or more correlation functions defining a correlation among different tags of a set of training examples, to produce a revised tag estimate, said training examples being audiovisual works of said defined type corresponding to manually-tagged audiovisual files; and

outputting a final result of the applying step as the set of tags for said input audiovisual file;

wherein the correlation-function application step applies said correlation functions selectively to the tags of said initial estimate by applying said correlation functions to tags having a correlation with the physical properties of audiovisual works of said defined type and not applying said correlation functions to tags that are poorly correlated with the physical properties of audiovisual works of said defined type.

2. The fully-automatic audiovisual-file tag production method according to claim 1, wherein the initial-estimate providing step comprises:

identifying training examples having a similar set of feature values to a set of feature values evaluated for said audiovisual work in the analyzing step; and

producing the initial estimate of the set of tags for the input audiovisual file by inference based on the tags of the training examples identified in the identifying step.

3. The fully-automatic audiovisual-file tag production method according to claim 2, configured to tag music files, wherein the analyzing step quantifies, as a feature value of the sets of feature values, at least one property, selected in a group consisting of harmonic properties and timbre, of a subject musical work represented by a subject music file, and the identifying step further determines whether the training examples are similar to the subject musical work based on whether the harmonic properties and/or timbre of the training examples is/are similar to the harmonic properties and/or timbre of the subject musical work quantified in the analyzing step.

4. The fully-automatic audiovisual-file tag production method according to claim 1, wherein the initial-estimate providing step associates a confidence measure with each tag of said initial estimate, and the correlation-function application step applies said correlation functions only to estimated tags associated with a confidence measure indicating a degree of confidence greater than a threshold level.

5. The fully-automatic audiovisual-file tag production method according to claim 1, wherein the correlation-function application step is applied iteratively and, after each iteration, there is provided the step of determining the best estimate so far of the value of each tag, wherein the first iteration applies said correlation functions to said initial estimate of the set of tags, and subsequent iterations apply said correlation functions to the best estimate so far of each tag.

6. A non-transitory computer readable medium storing a computer program having a set of instructions which, executed by a computer apparatus, cause the computer apparatus to perform a fully-automatic method of producing a set of tags for an input audiovisual file, the set of tags indicating values of a plurality of attributes of an audiovisual work of defined type represented by said audiovisual file, the method comprising:

analyzing properties of the audiovisual work represented by said input audiovisual file and evaluating a set of one or more features characterizing said properties of said audiovisual work;

providing an initial estimate of said set of tags by automatically converting said set of features evaluated in the analyzing step to said initial estimate based on first correlations between physical properties of audiovisual works of said defined type and tags applicable to audiovisual works of said defined type;

automatically applying, to the tags of said initial estimate, a set of one or more correlation functions defining a correlation among different tags of a set of training examples, to produce a revised tag estimate, said training examples being audiovisual works of said defined type corresponding to manually-tagged audiovisual files; and

outputting a final result of the applying step as the set of tags for said input audiovisual file;

wherein the correlation-function application step applies said correlation functions selectively to the tags of said initial estimate by applying said correlation functions to tags having a correlation with the physical properties of audiovisual works of said defined type and not applying said correlation functions to tags that are poorly correlated with the physical properties of audiovisual works of said defined type.

7. The non-transitory computer readable medium according to claim 6, wherein the initial-estimate providing step comprises:

identifying training examples having a similar set of feature values to a set of feature values evaluated for said audiovisual work in the analyzing step; and

producing the initial estimate of the set of tags for the input audiovisual file by inference based on the tags of the training examples identified in the identifying step.

8. The non-transitory computer readable medium according to claim 7, storing a computer program having a set of instructions which, executed by a computer apparatus, cause the computer apparatus to perform a fully-automatic music-file tag production method, wherein the analyzing step quantifies, as a feature value of the sets of feature values, at least one property, selected in a group consisting of harmonic properties and timbre, of a subject musical work represented by a subject music file, and the identifying step further determines whether the training examples are similar to the subject musical work based on whether the harmonic properties and/or timbre of the training examples is/are similar to the harmonic properties and/or timbre of the subject musical work quantified in the analyzing step.

9. The non-transitory computer readable medium according to claim 6, wherein the initial-estimate providing step associates a confidence measure with each tag of the initial estimate, and the correlation-function application step applies said correlation functions only to estimated tags associated with a confidence measure indicating a degree of confidence greater than a threshold level.

10. The non-transitory computer readable medium according to claim 6, wherein the correlation-function application step is applied iteratively and, after each iteration, there is provided the step of determining the best estimate so far of the value of each tag, wherein the first iteration applies said correlation functions to said initial estimate of the set of tags, and subsequent iterations apply said correlation functions to the best estimate so far of each tag.

11. A fully-automatic audiovisual-file-tagging system implemented by an information processing apparatus configured to output a set of tags for an input audiovisual file that indicate values of a plurality of attributes of an audiovisual work of a defined type represented by said audiovisual file, the system comprising:

an analyzing unit configured to analyze properties of the audiovisual work represented by said input audiovisual file and evaluate a set of one or more features that characterize said properties of said audiovisual work;

an initial estimate providing unit configured to provide an initial estimate of said set of tags by the automatic conversion of said set of features evaluated by the analyzing unit to said initial estimate based on first correlations between physical properties of audiovisual works of said defined type and tags applicable to audiovisual works of said defined type;

a correlation function application unit configured to automatically apply, to the tags of said initial estimate, a set of one or more correlation functions configured to define a correlation among different tags of a set of training examples, to produce a revised tag estimate, wherein said training examples are audiovisual works of said defined type that correspond to manually-tagged audiovisual files; and

a final result outputting unit configured to output a final result of the correlation function application unit as the set of tags for said input audiovisual file;

wherein the correlation function application unit is further configured to apply said correlation functions selectively to the tags of said initial estimate by applying said correlation functions to tags having a correlation with the physical properties of audiovisual works of said defined type and not applying said correlation functions to tags that are poorly correlated with the physical properties of audiovisual works of said defined type.

12. The fully-automatic audiovisual-file-tagging system according to claim 11, further comprising:

a memory storing attribute-value data for said set of training examples.

13. The fully-automatic audiovisual-file-tagging system according to claim 11, wherein the initial estimate providing unit comprises:

a training example identification unit configured to identify training examples having a similar set of feature values to a set of feature values evaluated for said input audiovisual work analyzed by the analyzing unit; and

an initial estimate of the attribute values unit configured to produce the initial estimate of the set of tags for the input audiovisual file by inference based on the tags of the training examples identified by the training example identification unit.

14. The fully-automatic audiovisual-file-tagging system according to claim 13, configured to tag music files, wherein the analyzing unit is further configured to quantify, as a feature value of the sets of feature values, at least one property, selected in a group consisting of harmonic properties and timbre, of a subject musical work represented by a subject music file, and the training example identification unit is further configured to determine whether the training examples are similar to the subject musical work based on whether the harmonic properties and/or timbre of the training examples is/are similar to the harmonic properties and/or timbre of the subject musical work quantified by the analyzing unit.

15. The fully-automatic audiovisual-file-tagging system according to claim 11, wherein the initial estimate providing unit is further configured to associate a confidence measure with each tag of the initial estimate, and the correlation function application unit is further configured to apply said correlation functions only to estimated tags associated with a confidence measure indicating a degree of confidence greater than a threshold level.

16. The fully-automatic audiovisual-file-tagging system according to claim 11, wherein the correlation function application unit is further configured to apply iterations and, after each iteration, is configured to determine the best estimate so far of the value of each tag, wherein a first iteration applies said correlation functions to said initial estimate of the set of tags, and subsequent iterations apply said correlation functions to the best estimate so far of each tag.