Method and system for classifying and locating media content

• US 6,735,583 B1
• Filed: 11/01/2000
• Issued: 05/11/2004
• Est. Priority Date: 11/01/2000
• Status: Expired due to Term

• Current Assignee: Getty Images Incorporated
• Original Assignee: Getty Images Incorporated
• Inventors: Bjarnestam, Anna, Kuhn, William G., Capel, Alan, Holm, Karina, Staczek, Jason
• Primary Examiner(s): Robinson, Greta
• Assistant Examiner(s): Black, Linh
• Application Number:
  US09/705,513
• Time in Patent Office: 1,287 Days
• Field of Search: 707/1-10, 707/100-104
• US Class Current: 1/1
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A computer-based method and system for classifying and locating media content is provided. In preferred embodiments, a dynamically modifiable, structured vocabulary system is used to drive a media classification and search system. An exemplary embodiment of the media classification and search system includes media content unit classification services, a media content unit search engine, and one or more data repositories, including typically a media content unit collection, metadata, the vocabulary, and a term-to-media content reverse index. The structured vocabulary provides a classification system, whose structure and relationships can be altered independently of the media content being classified, thus enabling tuning of the system without needing to reclassify data. The use of the structured vocabulary also increases the rate of success and efficiency in matching search requests to locate media content.

Claims

1. A method in a computer system for classifying and locating a plurality of media content units using a structured vocabulary stored in a data repository, the structured vocabulary having a plurality of terms ordered such that the relationship of each term to each other term is inherent in the ordering, comprising:

for each media content unit,

determining a term within the structured vocabulary that represents a characteristic of the media content unit; and

associating an identifier of the determined term with the media content unit so that the media content unit is characterized by the determined term;

receiving a request to locate a media content unit based upon a search term; and

using the structured vocabulary to determine a collection of media content units that are characterized by the search term.

2. The method of claim 1, further comprising modifying the ordering of one of the determined terms within the structured vocabulary so that its relationship to other terms is altered, the modification occurring in a manner that does not affect prior associations between the one term and a media content unit.

3. The method of claim 1 wherein the ordering is hierarchical such that terms located further down in the hierarchy provide more specificity.

4. The method of claim 1 wherein the ordering includes terms that are subordinated to a plurality of other terms.

5. The method of claim 1 wherein the determining of a term and the associating of an identifier of the determined term with a media content unit reflects an objective classification of the media content unit.

6. The method of claim 1 wherein the determining of a term and the associating of an identifier of the determined term with a media content unit reflects a subjective classification of the media content unit.

7. The method of claim 1 wherein the search term that is used to locate a media content unit is specified in a language other than the language of a determined term of the structured vocabulary that was used to characterize the media content unit.

8. The method of claim 1 wherein a term is ordered by implicitly associating the term with a plurality of other terms.

9. The method of claim 1 wherein the search term is normalized to convert search term variants to a same search term.

10. The method of claim 1 wherein the structured vocabulary comprises terms in a second language that correspond to terms expressed in a first language of the structured vocabulary.

11. The method of claim 10 wherein the association of the identifier of a term in the first language with a media content unit results in the media content unit being automatically characterized by the corresponding term in the second language.

12. The method of claim 1 wherein ambiguous search terms are disambiguated by presenting the relationship of the search term to a plurality of other terms.

13. The method of claim 12 wherein the disambiguation of search terms may be optionally activated.

14. The method of claim 12 wherein the disambiguated search term is used to locate media content units that are characterized by the disambiguated search term.

15. The method of claim 1, further comprising, for each term of the structured vocabulary, creating an index from the term to a collection of media content units that are characterized by the term.

16. The method of claim 15, further comprising using the created index to efficiently determine the collection of media content units that are characterized by the search term.

17. The method of claim 1 wherein the media content units comprise digital images.

18. The method of claim 1 wherein the media content units comprise sound recordings.

19. The method of claim 1 wherein the media content units comprise text.

20. A structured vocabulary stored in a computer memory usable with a computer system, comprising:

a plurality of media content units; and

a plurality of identifiers of terms, wherein the terms are ordered in the structured vocabulary according to inherent relationships between the terms, such that each media content unit is associated with an identifier of at least one term, and wherein each media content unit can be determined by matching a requested term with an identifier of a corresponding term.

21. The structured vocabulary of claim 20 wherein each term is stored in a vocabulary unit.

22. The structured vocabulary of claim 21 wherein the vocabulary unit stores synonyms of the term stored in the vocabulary unit.

23. The structured vocabulary of claim 20, wherein each identifier of a term also identifies synonyms of that term.

24. The structured vocabulary of claim 23, wherein a synonym includes a translation of the term into another language.

25. The structured vocabulary of claim 20 wherein each term is associated with a particular media content unit by storing an identifier of the term in a metadata structure associated with the media content unit.

26. The structured vocabulary of claim 25 wherein the metadata structure is used to determine the media content unit when a request term is received by the computer system that specifies the term referred to by the metadata structure.

27. The structured vocabulary of claim 20, further comprising a reverse index stored in the computer memory that stores, for each term, an identifier of each media content unit that is associated with the identifier of that term.

28. The structured vocabulary of claim 27 wherein the reverse index is used to efficiently determine a set of media content units associated with a requested term.

29. The structured vocabulary of claim 20 wherein the plurality of terms comprises terms in a plurality of languages.

30. The structured vocabulary of claim 29 wherein two term in different languages that share a common meaning can be used interchangeably to locate a media content unit associated with the identifier of one of the two terms.

31. The structured vocabulary of claim 20 wherein the media content units comprise digital images.

32. The structured vocabulary of claim 20 wherein the media content units comprise sound recordings.

33. The structured vocabulary of claim 20 wherein the media content units comprise text.

34. A media classification and retrieval system comprising:

a structured vocabulary stored in a data repository and having a plurality of terms ordered such that the relationship of each term to each other term is inherent in the ordering;

a plurality of media content units wherein each media content unit is associated with an identifier of a term of the structure vocabulary that represents a characteristic of the media content unit; and

a search engine that receives a request to locate a media content unit based upon a search term and that uses the structured vocabulary and the associations between media content units and identifiers of terms in the structured vocabulary to determine a collection of media content units that are characterized by the search term.

35. The system of claim 34 wherein the ordering of one of the terms that represents a characteristic of a media content unit is modified so that its relationship to other terms is altered, the modification occurring in a manner that does not affect prior associations between the one term and a media content unit.

36. The system of claim 34 wherein the ordering of the terms of the structured vocabulary is hierarchical such that terms located further down in the hierarchy provide more specificity.

37. The system of claim 34 wherein the ordering of the terms of the structured vocabulary includes terms that are subordinated to a plurality of other terms.

38. The system of claim 34 wherein a term and an identifier of the term are associated with a media content unit according to an objective classification.

39. The system of claim 34 wherein a term and an identifier of the term are associated with the media content unit according to a subjective classification.

40. The system of claim 34 wherein the search engine accepts a search term that is specified in a language other than the language of the term of the structured vocabulary whose identifier was associated with the media content unit.

41. The system of claim 34 wherein a term is ordered by implicitly associating the term with a plurality of other terms.

42. The system of claim 34 wherein the search engine normalizes the search term to convert search term variances to a same search term.

43. The system of claim 34 wherein the structured vocabulary comprises terms in a second language that correspond to terms expressed in a first language of the structured vocabulary.

44. The system of claim 43 wherein the association of an identifier of a term in the first language with a media content unit results in the media content unit being automatically characterized by a corresponding term in the second language.

45. The system of claim 34 wherein the search engine disambiguates ambiguous search terms by presenting the relationship of the search term to a plurality of other terms.

46. The system of claim 45 wherein the search engine may optionally activate the disambiguation of search terms.

47. The system of claim 45 wherein the disambiguated search term is used to locate media content units.

48. The system of claim 34, further comprising an index that associates a collection of media content units with a term that characterizes each of the media content units of the collection.

49. The system of claim 48 wherein the search engine uses the index to efficiently determine the collection of media content units that are characterized by the search term.

50. The system of claim 34 wherein the media content units comprise digital images.

51. The system of claim 34 wherein the media content units comprise sound recordings.

52. The system of claim 34 wherein the media content units comprise text.

81. The method of claim 50, wherein the search term is different from the term used to classify the indicated media content unit.

82. The method of claim 81 wherein the term used to classify the indicated media content unit is of a language different than the search term.

53. A method in a computer system for reclassifying media content using a structured vocabulary stored in the computer system, the computer system having a plurality of stored media content units that have been each classified according to current terms of the structured vocabulary comprising:

receiving a descriptive term; and

based upon the received descriptive term, modifying the structured vocabulary without destroying the relationships of the already classified media content units to current terms of the structured vocabulary.

54. The method of claim 53 wherein modifying the structured vocabulary includes storing an identifier of the descriptive term in the structured vocabulary and whereby the location of the stored identifier in the structured vocabulary indicates the relationship of the descriptive term to other terms in the vocabulary.

55. The method of claim 53 wherein modifying the structured vocabulary includes changing the ordering of terms within the structured vocabulary.

56. The method of claim 53 wherein modifying the structured vocabulary includes adding a synonym of an existing term to the structured vocabulary.

57. The method of claim 56 wherein adding the synonym of the existing term adds a term in a second language to the structured vocabulary.

58. The method of claim 53 wherein modifying the structured vocabulary includes deleting a term from the structured vocabulary.

59. The method of claim 53 wherein modifying the structured vocabulary includes deleting a synonym of an existing term from the structured vocabulary.

60. The method of claim 53, further comprising retrieving a collection of media content units based upon a search term.

61. The method of claim 53, further comprising, upon receiving the descriptive term, determining whether the received term is ambiguous.

62. The method of claim 61 wherein the determination of ambiguity is made by examining whether the received term already exists in the structured vocabulary.

63. The method of claim 53 wherein modifying the structured vocabulary includes providing a set of choices that convey multiple meanings for a current ambiguous term thereby allowing the current term to be disambiguated.

64. A media classification system comprising:

a structured vocabulary stored in a data repository having a plurality of terms;

a plurality of media content units stored in a data repository, wherein each media content unit has been classified according to the plurality of terms of the structured vocabulary; and

a classification engine that receives a descriptive term and, based upon the received descriptive term, modifies the structured vocabulary without destroying the relationships of the already classified media content units to the plurality of terms of the structured vocabulary.

65. The system of claim 64 wherein the classification engine modifies the structured vocabulary to incorporate the descriptive term by storing an identifier of the descriptive term in the structured vocabulary and whereby the relative location of the stored identifier in the structured vocabulary indicates the relationship of the descriptive term to other terms in the vocabulary.

66. The system of claim 64 wherein the classification engine modifies the structured vocabulary by changing the ordering of terms within the structured vocabulary.

67. The system of claim 64 wherein the classification engine modifies the structured vocabulary by adding a synonym of an existing term to the structured vocabulary.

68. The system of claim 67 wherein the classification engine adds the synonym of the existing term by adding a term in a second language to the structured vocabulary.

69. The system of claim 64 wherein the classification engine modifies the structured vocabulary by deleting a term from the structured vocabulary.

70. The system of claim 64 wherein the classification engine modifies the structured vocabulary by deleting a synonym of an existing term from the structured vocabulary.

71. The system of claim 64, further comprising a retrieval engine that retrieves a collection of media content units by locating a search term in the modified structured vocabulary.

72. A computer-readable memory medium containing instructions for controlling a computer processor to reclassify media content using a structured vocabulary stored in a data repository and using a plurality of media content units stored in a data repository that have been each classified according to a plurality of terms of the structured vocabulary, by:

receiving a descriptive term; and

based upon the received descriptive term, modifying the stored structured vocabulary without destroying the relationships of the already classified media content units to the plurality of terms of the structured vocabulary.

73. The computer-readable memory medium of claim 72 wherein modifying the structured vocabulary includes storing an identifier of the descriptive term in the structured vocabulary and whereby the location of the stored identifier in the structured vocabulary indicates the relationship of the descriptive term to other terms in the vocabulary.

74. The computer-readable memory medium of claim 72 wherein modifying the structured vocabulary includes changing the ordering of terms within the structured vocabulary.

75. The computer-readable memory medium of claim 72 wherein modifying the structured vocabulary includes adding a synonym of an existing term to the structured vocabulary.

76. The computer-readable memory medium of claim 75 wherein adding the synonym of the existing term adds a term in a second language to the structured vocabulary.

77. The computer-readable memory medium of claim 72 wherein modifying the structured vocabulary includes deleting a term from the structured vocabulary.

78. The computer-readable memory medium of claim 72 wherein modifying the structured vocabulary includes deleting a synonym of an existing term from the structured vocabulary.

79. The computer-readable memory medium of claim 72 further comprising instructions that control a computer processor to retrieve a collection of media content units based upon a search term.

80. A method in a first computer system for retrieving a media content unit from a second computer system having a plurality of media content units that have been classified according to terms of a structured vocabulary, wherein the terms are ordered such that the relationship of each term to each other term is inherent in the ordering, comprising:

sending a request for a media content unit, the request specifying a search term; and

receiving an indication of a media content unit that corresponds to the specified search term, wherein the search term is located within the structured vocabulary and is used to determine a media content unit that corresponds to the search term.

83. The method of claim 80, further comprising:

receiving from the second computer system an indication of multiple instances of the search term in the structured vocabulary; and

sending to the second computer system a revised request that specifies one of the multiple instance of the search term.

84. The method of claim 80 wherein the received media content unit is a digital image.

85. The method of claim 80 wherein the received media content unit is a sound recording.

86. The method of claim 80 wherein the received media content unit is a text record.

87. The method of claim 80 wherein the first and second computer system are the same computer system.

88. The method of claim 80 wherein the first and second computer systems are physically distinct.

89. A method in a computer system for rapidly locating a plurality of media content units that have been classified using a plurality of ordered terms in a structured vocabulary by associating each media content unit with at least one of the ordered terms, the terms ordered such that the relationship of each term to each other term is inherent in the ordering, the computer system having a reverse index that indexes the media content units by each associated term, comprising:

receiving a request to locate a media content unit using a search term; and

using the search term as an index into the reverse index to locate the media content units associated by the search term.

90. The method of claim 89 wherein the terms of the structured vocabulary are ordered in a hierarchical fashion.

91. The method of claim 89 wherein each media content unit has a corresponding metadata structure and wherein a reference to each of the ordered terms of the structured vocabulary that is associated with the media content unit is stored in the metadata structure corresponding to the associated media content unit.

92. The method of claim 91 wherein the metadata structures are used to construct the reverse index.

93. The method of claim 89 wherein the search term is normalized to reduce variations of the different search terms to a same term.

94. The method of claim 89 wherein the media content units comprise digital images.

95. The method of claim 89 wherein the media content units comprise sound recordings.

96. The method of claim 89 wherein the media content units comprise text.

97. A media content retrieval system comprising:

a structured vocabulary stored in a data repository and having a plurality of terms ordered such that the relationship of each term to each other term is inherent in the ordering;

a plurality of media content units stored in a data repository, wherein each media content unit has been classified by associating an identifier of a descriptive term with the media content unit; and

a search engine having a reverse index that indexes the media content units by terms, the search engine being configured to receive a request to locate a media content unit using a search term and being configured to use the search term as an index into the reverse index to locate the media content units associated with an identifier of the search term.

98. The system of claim 97 wherein the terms of the structured vocabulary are ordered in a hierarchical fashion.

99. The system of claim 97 wherein each media content unit has a corresponding metadata structure and wherein a reference to each of the ordered terms of the structured vocabulary that is associated with the media content unit is stored in the corresponding metadata structure corresponding to the associated media content unit.

100. The system of claim 99 wherein the metadata structures are used to construct the reverse index.

101. The system of claim 99 wherein each metadata structure contains data in addition to a reference to a term in the structured vocabulary.

102. The system of claim 97 wherein the search term is normalized to a canonical form.

103. The system of claim 97 wherein the media content unit comprises digital images.

104. The system of claim 97 wherein the media content units comprise sound recordings.

105. The system of claim 97 wherein the media content units comprise text.

106. The system of claim 97 wherein the reverse index indexes a particular media content unit by a general term in addition to indexing the particular media content unit by a term that indicates a more specific meaning of the general term.

107. A computer-readable memory medium containing instructions for controlling a computer processor of a computer system to efficiently retrieve media content that has been classified using a plurality of ordered terms in a structured vocabulary by associating each media content unit with at least one of the ordered terms, the computer system having a reverse index that indexes the media content units by each associated term, by:

receiving a request to locate a media content unit using a search term; and

using the search term as an index into the reverse index to locate the media content units associated by the search term.

108. The computer-readable memory medium of claim 107 wherein the terms of the structured vocabulary are ordered in a hierarchical fashion.

109. The computer-readable memory medium of claim 107 wherein each media content unit has a corresponding metadata structure and wherein a reference to each of the ordered terms of the structured vocabulary that is associated with the media content unit is stored in the metadata structure corresponding to the associated media content unit.

110. The computer-readable memory medium of claim 109 wherein the metadata structures are used to construct the reverse index.

111. The computer-readable memory medium of claim 107 wherein the search term is normalized.

112. The computer-readable memory medium of claim 107 wherein the media content units comprise digital images.

113. The computer-readable memory medium of claim 107 wherein the media content units comprise sound recordings.

114. The computer-readable memory medium of claim 107 wherein the media content units comprise text.