Business method and apparatus for employing induced multimedia classifiers based on unified representation of features reflecting disparate modalities

• US 20030004966A1
• Filed: 06/18/2001
• Published: 01/02/2003
• Est. Priority Date: 06/18/2001
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Haas, Norman, Zhang, Tong, Bolle, Rudolf M., Oles, Frank J.
• Granted Patent: US 6,993,535 B2
• Time in Patent Office: Days
• US Class Current: 707/104.1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/637 Administration of user profiles, e.g. generation, initialization, adaptation or distribution
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 16/735 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

This invention is a business system and method to perform categorization (classification) of multimedia items and to make business decisions based on the categorization of the item. The multimedia items are comprised of a multitude of disparate information sources, in particular, visual information and textual information. Classifiers are induced based on combining textual and visual feature vectors. Textual features are the traditional ones. Visual features include, but are not limited to, color properties of key intervals and motion properties of key intervals. The visual feature vectors are determined in such a fashion that the vectors are sparse. The text and the visual representation vectors are combined in a systematic and coherent fashion. This vector representation of a media item lends itself to well-established learning techniques and can be used for multimedia item categorization. The resulting business system, subject of this invention, can be used for many purposes. An example here are enforcement of copyright, trademark, intellectual property, parental guidance and common decency restrictions. Other uses are multimedia item classifier to determine routing of incoming items or building user profiles based on user multimedia preferences.

Claims

1. A business method comprising the steps of:

monitoring one or more multimedia items accessed by a user, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities;

creating a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities;

concatenating the visual feature vectors and the textual feature vectors into a unified feature vector;

categorizing each of the multimedia items by categorizing the respective unified feature vector; and

assembling a user profile based on the categorization.

2. A business method, as in claim 1, further comprising the step of:

using the user profile to match one or more multimedia items stored in one or more databases.

3. A business method, as in claim 2, where one or more of the databases are part of a computer that is connected to one or more networks.

4. A business method, as in claim 3, where the networks include any one or more of the following:

an Internet, an intranet, an extranet, a corporate network, a government network, a infrared network, and a radio frequency network.

5. A business method, as in claim 1, where the categories include any one or more of the following:

a product, a service, an interest, a retail item, a hobby, a food item, an item of clothing, a travel package, a vacation destination, a financial product, a business partner, a business interest, a medical product, a commercial, and a social interest.

6. A business method, as in claim 1, where the category includes any one or more of the following services:

consulting, legal, real estate, medical, technical, physical training, diet, cosmetic, fashion, governmental, automotive, design, architecture, personal assistants, games, on-line games of chance, dating services, and landscaping.

7. A business method comprising the steps of:

scanning one or more multimedia items in a database, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities;

creating a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities;

concatenating the visual feature vectors and the textual feature vectors into a unified feature vector;

categorizing each of the multimedia items by categorizing the respective unified feature vector; and

creating one or more indices of the database based on the categorization.

8. A business method, as in claim 7, where the database resides on any one or more of the following:

a network server, a web site, a personal computer, a server farm, and a network disk array.

9. A business method, as in claim 7, further comprising the step of:

making a business decision based on the classifications.

10. A business method, as in claim 9, where the categorization is used to organize a collection of the multimedia items into a database where the multimedia items are retrievable based on an annotation of the classification.

11. A business method, as in claim 10, where the multimedia items that are retrieved are used as a response to a query of a search engine.

12. A business method, as in claim 9, where the multimedia item is a multimedia e-mail and the multimedia e-mail is routed based on one or more of the categories assigned to the multimedia e-mail.

13. A business method, as in claim 12, where the multimedia e-mail is routed to any one or more of the following:

a sender, a folder, a person, a personal folder, a corporate folder, and a corporate department.

14. A business method, as in claim 13, where the multimedia e-mail is multiplied before being routed.

16. A business method, as in claim 15, where the multimedia items are digital multimedia copies and the feature vectors represent restricted multimedia items and the comparison determines if the digital multimedia copies are similar to the restricted multimedia items and the decision is that the digital multimedia copies are subject to a second restriction.

17. A business method, as in claim 16, where the second restriction includes any one or more of the following:

a copyright restriction, a trademark restriction, intellectual property restriction, parental guidance restriction, common decency restrictions, user-defined restrictions.

18. A business method, as in claim 16, where the digital multimedia copies are accessible over the Internet.

19. A business method, as in claim 15, where the comparison indicates a degree of similarity and the decision is that the multimedia item contains a known content.

15. A business method comprising the steps of:

scanning one or more multimedia items in a database, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities;

creating a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities;

concatenating the visual feature vectors and the textual feature vectors into a unified feature vector;

categorizing each of the multimedia items by categorizing the respective unified feature vector;

comparing one or more of the unified feature vectors to one or more other feature vectors; and

making a decision based on the comparison.