Method and system for building an integrated user profile

• US 9,564,123 B1
• Filed: 05/05/2015
• Issued: 02/07/2017
• Est. Priority Date: 05/12/2014
• Status: Active Grant

• Current Assignee: Soundhound AI IP LLC
• Original Assignee: SoundHound Incorporated
• Inventors: Mont-Reynaud, Bernard, Huang, Jun, Lokeswarappa, Kiran Garaga, Gedalius, Joel
• Primary Examiner(s): Hang, Vu B
• Application Number:
  US14/704,833
• Time in Patent Office: 644 Days
• Field of Search: 704/9, 704/231, 704/246, 704/243, 704/244, 704/257
• US Class Current: 1/1
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 30/0251 Targeted advertisements
• CPC Class Codes: G06Q 30/0276 Advertisement creation
• CPC Class Codes: G06Q 30/0277 Online advertisement
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals
• CPC Class Codes: H04L 67/306 User profiles

Abstract

A system and method are provided for adding user characterization information to a user profile by analyzing user'"'"'s speech. User properties such as age, gender, accent, and English proficiency may be inferred by extracting and deriving features from user speech, without the user having to configure such information manually. A feature extraction module that receives audio signals as input extracts acoustic, phonetic, textual, linguistic, and semantic features. The module may be a system component independent of any particular vertical application or may be embedded in an application that accepts voice input and performs natural language understanding. A profile generation module receives the features extracted by the feature extraction module and uses classifiers to determine user property values based on the extracted and derived features and store these values in a user profile. The resulting profile variables may be globally available to other applications.

Claims

1. A method for assigning values to user profile properties based on user speech analysis, comprising:

receiving extracted features from a feature extraction module, including at least acoustic features, the extracted features representing user speech observed over a first time period;

deriving features by executing a function that uses one or more of the extracted features as input, wherein at least one of the derived features is derived from an extracted acoustic feature;

storing, using a processor, derived features;

computing, using a processor, extended features as a function of one or more of;

the extracted features;

the derived features; and

statistics based on the extracted features and the derived features received during a second time period that is longer than the first time period;

deriving a value of a user profile property by executing a classifier that uses the computed extended features as input, the classifier trained on labeled extended features for the user profile property; and

storing, by a processor, the value assigned to the user profile property in a user profile.

2. The method of claim 1, wherein the classifier is trained using one or more of:

a pattern recognition technique; and

a machine learning technique.

3. The method of claim 1, wherein the user profile property is one of:

age;

gender;

accent;

reading level;

education level;

English language proficiency; and

socio-economic status (SES).

4. The method of claim 1, wherein a second value of a second user profile property is used as a constraint on the value of the user profile property.

5. The method of claim 1, wherein extracted features are received from a plurality of feature extractors.

6. The method of claim 1, wherein the extracted features include a plurality of:

a cepstrum, a spectrogram, and a phoneme lattice.

19. The method of claim 1, wherein the received extracted features, representing user speech observed over a first time period further includes at least one of:

phonetic features, textual features, linguistic features, and semantic features.

7. A system for assigning values to user profile properties based on user speech analysis, comprising:

a processor; and

a non-transitory computer-readable storage medium storing instructions, which when executed by the processor, cause the processor to;

receive extracted features, including at least acoustic features, the extracted features representing user speech observed over a first time period;

derive features by executing a function that uses one or more of the extracted features as input, wherein at least one of the derived features is derived from an extracted acoustic feature;

store derived features;

compute extended features as a function of one or more of;

the extracted features;

the derived features; and

statistics based on the extracted features and the derived features received during a second time period that is longer than the first time period;

derive a value of a user profile property by executing a classifier that uses the computed extended features as input, the classifier trained on labeled extended features for the user profile property; and

store the value assigned to the profile property in a user profile.

8. The system of claim 7, wherein the classifier is trained using one or more of:

a pattern recognition technique; and

a machine learning technique.

9. The system of claim 7, wherein the user profile property is one of:

age;

gender;

accent;

reading level;

education level;

English language proficiency; and

socio-economic status (SES).

10. The system of claim 7, wherein a second value of a second user profile property is used as a constraint on the value of the user profile property.

11. The system of claim 7, wherein extracted features are received from a plurality of feature extractors.

12. A non-transitory computer-readable storage medium storing instructions for assigning values to user profile properties based on user speech analysis, the instructions, which when executed by a processor, cause the processor to:

receive extracted features, including at least acoustic features, the extracted features representing user speech observed over a first time period;

derive features by executing a function that uses one or more of the extracted features as input, wherein at least one of the derived features is derived from an extracted acoustic feature;

store derived features;

compute extended features as a function of one or more of;

the extracted features;

the derived features; and

statistics based on the extracted features and the derived features received during a second time period that is longer than the first time period;

derive the value of a user profile property by executing a classifier that uses the computed extended features as input, the classifier trained on labeled extended features for the user profile property; and

store the value assigned to the profile property in a user profile.

13. The non-transitory computer-readable storage medium of claim 12, wherein the classifier is trained using one or more of:

a pattern recognition technique; and

a machine learning technique.

14. The non-transitory computer-readable storage medium of claim 12, wherein the user profile property is one of:

age;

gender;

accent;

reading level;

education level;

English language proficiency; and

socio-economic status (SES).

15. The non-transitory computer-readable storage medium of claim 12, wherein a second value of a second user profile property is used as a constraint on the value of the user profile property.

16. The non-transitory computer-readable storage medium of claim 12, wherein extracted features are received from a plurality of feature extractors.

17. A method to determine user profile property values based on textual input, comprising:

receiving from a text-based natural language application a textual input entered by a user;

creating a parse tree by processing the textual input against a grammar;

deriving linguistic features from the parse tree;

computing, using a processor, extended features as a function of one or more of;

the linguistic features; and

statistics based on the derived linguistic features;

deriving the value of a user profile property by executing a classifier that uses the computed extended features as input, the classifier trained on labeled extended features for the user profile property; and

storing the value assigned to the user profile property in a user profile.

18. The method of claim 17, wherein the derived linguistic features include an indication of use of grammar or vocabulary.