Systems and methods for inferring gender by fusion of multimodal content

• US 9,684,852 B2
• Filed: 08/02/2016
• Issued: 06/20/2017
• Est. Priority Date: 06/29/2015
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cao, Liangliang, Merler, Michele, Smith, John R.
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/226,789
• Publication Number:
  US 20160379086A1
• Time in Patent Office: 322 Days
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

A method and systems are provided. A system includes a set of visual and textual classifiers for recognizing semantic concepts in a set of images and assigning semantic scores for the images to predict a gender of a user, and performing gender prediction from visual content and textual content in the images to respectively generate visual-based gender predictions and textual-based gender predictions. The system further includes a multimodal information fusion device for combining, using multimodal information fusion, the visual-based gender predictions, the textual-based gender predictions, and the semantic scores to infer a gender of a user.

Claims

1. A system, comprising:

a set of visual and textual classifiers for recognizing semantic concepts in a set of images and assigning semantic scores for the images to predict a gender of a user, and performing gender prediction from visual content and textual content in the images to respectively generate visual-based gender predictions and textual-based gender predictions; and

a multimodal information fusion device for combining, using multimodal information fusion, the visual-based gender predictions, the textual-based gender predictions, and the semantic scores to infer a gender of a user.

2. The system of claim 1, wherein the semantic concepts comprise visual-based semantic concepts and textual-based semantic concepts.

3. The system of claim 1, wherein at least one of the visual classifiers and at least one of the textual classifiers generate semantic distributions from the semantic scores, and wherein some of the visual-based gender predictions and the textual-based gender predictions are determined from the semantic distributions.

4. The system of claim 1, wherein at least one of the visual classifiers and at least one of the textual classifiers generate semantic distributions from the semantic scores, wherein some of the visual-based gender predictions and the textual-based gender predictions are determined from aggregations of the semantic distributions.

5. The system of claim 1, wherein said multimodal information fusion device performs filtered fusion by selecting which multimodal gender information to use directly and which of the multimodal gender information to aggregate, to infer the gender of the user, from among the visual-based gender predictions, the textual-based gender predictions, and the semantic scores.

6. The system of claim 1, wherein said multimodal information fusion device further comprises a confidence checker for performing confidence checking on at least some of the semantic scores to influence an inference of the gender of the user.

7. The system of claim 6, wherein the confidence checking is performed separately by a first confidence check on the at least some of the semantic scores relating to the visual content and by a second confidence check on the at least some of the semantic scores relating to the textual content, and the inference of the gender of the user is influenced differently based on an agreement or a non-agreement between the first and the second confidence checks.

8. The system of claim 1, wherein the visual content comprises profile colors and a background scene in a profile picture of the user.

9. A system for inferring gender, comprising:

a set of visual classifiers, each for recognizing visual content from a set of images associated with a user, assigning respective visual-based prediction confidence scores for each of the images based the recognized visual-based content, and at least one for generating a visual-based gender prediction of the user based on the respective visual-based prediction confidence scores for each of the images;

a set of textual classifiers, each for recognizing textual content from the set of images, assigning respective textual-based prediction confidence scores for each of the images based the recognized textual content, and at least one for generating a visual-based gender prediction of the user based on the respective visual-based prediction confidence scores for each of the images; and

a multimodal information fusion device for inferring a gender of a user selectively from the visual-based gender predictions, the textual-based gender predictions, the visual-based prediction confidence scores and the textual-based prediction confidence scores.

10. The system of claim 9, wherein the at least one of the visual classifiers determines respective semantic distributions across the set of images for different ones of the recognized visual content, each of the respective semantic distribution based on an aggregation of the respective visual-based prediction confidence scores for a same one of the recognized visual content for each of the images.

11. The system of claim 9, wherein the at least one of the visual classifiers generates the visual-based gender prediction based on the respective semantic distributions.