Multi-modal gender recognition including depth data

• US 8,675,981 B2
• Filed: 06/11/2010
• Issued: 03/18/2014
• Est. Priority Date: 06/11/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Kipman, Alex Aben-Athar
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US12/813,675
• Publication Number:
  US 20110307260A1
• Time in Patent Office: 1,376 Days
• Field of Search: None
• US Class Current: 382/260
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Gender recognition is performed using two or more modalities. For example, depth image data and one or more types of data other than depth image data is received. The data pertains to a person. The different types of data are fused together to automatically determine gender of the person. A computing system can subsequently interact with the person based on the determination of gender.

Claims

1. A method for a data processing system to interact with a user, comprising:

receiving depth data about a person;

receiving one or more types of data other than depth data about the person;

automatically determining gender of the person based on the depth data and the one or more types of data other than depth data;

assigning weights to the depth data and the one or more other types of data based on quality of the data, the automatically determining gender is further based on the assigned weights; and

performing an action based on the determination of gender.

2. The method of claim 1, wherein:

the receiving one or more types of data other than depth data comprises receiving audio data for the person'"'"'s voice.

3. The method of claim 1, wherein:

the receiving one or more types of data other than depth data comprises receiving a visual image, infra-red image or thermal image of the person.

4. The method of claim 1, wherein the automatically determining gender comprises:

extracting a first set of one or more features from the depth data and making a first determination of gender based on the first set of one or more features;

extracting a second set of one or more features from the depth data and making a second determination of gender based on the second set of one or more features;

extracting a third set of one or more features from a different type of data and making a third determination of gender based on the third set of one or more features, the different type of data is one of the one or more types of data other than depth data;

extracting a fourth set of one or more features from the different type of data and making a fourth determination of gender based on the fourth set of one or more features; and

forming a conclusion of gender based on the first determination of gender, the second determination of gender, the third determination of gender and the fourth determination of gender.

5. The method of claim 4, further comprising:

identifying a first weight to the depth data and a second weight to the different type of data;

assigning the first weight to the first determination of gender;

assigning the first weight to the second determination of gender;

assigning the second weight to the third determination of gender; and

assigning the second weight to the fourth determination of gender, the forming a conclusion includes using the determinations of gender and assigned weights.

6. The method of claim 5, further comprising:

normalizing the depth data by transforming a depth image, the transforming the depth image changes the orientation of the person in the depth image.

7. The method of claim 6, wherein:

the making a first determination of gender includes assigning the first set of one or more features to a specific Gaussian distribution of a plurality of Gaussian distributions based on likelihood of belonging to the specific Gaussian distribution;

the assigning the first weight to the first determination of gender includes multiplying the first weight by the likelihood of belonging to the specific Gaussian distribution to create a product for the first determination of gender; and

the combining the determinations of gender and assigned weights includes adding the product for the first determination of gender to products for the second, third and fourth determinations of gender to create a composite score and determining gender of the person based on the composite score.

8. The method of claim 1, wherein the automatically determining gender comprises:

filtering the depth data to identify data for a first feature;

filtering the one or more types of data to identify data for other features; and

identifying gender of the person based on the data for the first feature and the data for the other features.

9. The method of claim 1, wherein the automatically determining gender comprises:

plugging in the depth data and the one or more types of data other than depth data into a set of one or more equations that make one decision of gender.

10. The method of claim 1, wherein the performing an action based on the determination of gender comprises:

creating and displaying an avatar having a gender matching the determined gender of the person.

11. The method of claim 1, wherein the performing an action based on the determination of gender comprises:

choosing a specific voice command library from a plurality of voice command libraries based on the determination of gender;

receiving a voice command; and

interpreting the received voice command using the specific voice command library.

12. One or more processor readable storage devices having processor readable code stored thereon, the processor readable code programs the one or more processors, the processor readable code comprising:

code that extracts depth features;

code that extracts one or more other features;

code, in communication with the code that extracts depth features, that classifies gender based on at least one depth feature;

code, in communication with the code that extracts one or more other features, that classifies gender based on at least one feature other than depth;

code which makes a gender determination based on output from the code that classifies gender based on at least one depth feature and the code that classifies gender based on at least one feature other than depth; and

code that estimates reliability of depth data, visual image data and voice data and determines weights for the depth data, visual image data and voice data based on the estimated reliability;

the code which makes the gender determination based on output from the code that classifies gender based on at least one depth feature and the code that classifies gender based on at least one feature other than depth makes its gender determination based on the weights for the depth data, visual image data and voice data.

13. The one or more processor readable storage devices of claim 12, wherein:

the code that extracts one or more other features comprises code that extracts visual image features and code that extracts voice features; and

the code that classifies gender based on at least one feature other than depth comprises code that that classifies gender based on at least one visual image feature and code that that classifies gender based on at least one visual image feature.

14. The one or more processor readable storage devices of claim 12, wherein:

the code that classifies gender based on at least one depth feature includes first code that makes a first gender classification based on shape of a subject in a depth image and second code that makes a separate second gender classification based on curvature of a surface of the subject in the depth image.

15. The one or more processor readable storage devices of claim 14, wherein:

the code that classifies gender based on at least one feature other than depth includes third code that makes a separate third gender classification based on pitch of the subject'"'"'s voice and fourth code that makes a separate fourth gender classification based on an image of the subject; and

the code which makes the gender determination based on output from the code that classifies gender based on at least one depth feature and the code that classifies gender based on at least one feature other than depth makes a final gender decision based on the first gender classification, the second gender classification, the third gender classification and the fourth gender classification.

16. A computing apparatus, comprising:

a depth sensor, the depth sensor acquires depth data about a person;

a different sensor that is different from the depth sensor, the different sensor acquires one or more types of data other than depth data about the person;

and

a processor in communication with the depth sensor and the different sensor, the processor assigns weights to the depth data and the one or more other types of data based on quality of the data, the processor determines gender of the person based on the depth data and the one or more types of data other than depth data, the processor determines gender of the person based on the assigned weights, the processor performs an action based on the determination of gender.

17. The computer of claim 16, wherein:

the computing apparatus further comprises an image sensor in communication with the processor;

the different sensor is an audio sensor;

the processor determines gender of the person based on data from the depth sensor, the image sensor and the audio sensor.

18. The computer of claim 17, wherein:

the processor automatically tracks motion of the person whose gender is being determined using data from the depth sensor; and

the processor performs the action based on the determination of gender in response to the tracking motion.

19. The computer of claim 16, wherein:

the performing an action based on the determination of gender comprises creating and displaying an avatar having a gender matching the determined gender of the person.

20. The computer of claim 16, wherein:

the one or more types of data other than depth data comprises audio data for the person'"'"'s voice.