Method and system for building an integrated user profile

• US 10,311,858 B1
• Filed: 12/20/2016
• Issued: 06/04/2019
• Est. Priority Date: 05/12/2014
• Status: Active Grant

• Current Assignee: Soundhound AI IP LLC (SoundHound AI, Inc. (f/k/a Archimedes Tech SPAC Partners Co.))
• Original Assignee: SoundHound, Inc. (SoundHound AI, Inc. (f/k/a Archimedes Tech SPAC Partners Co.))
• Inventors: Mont-Reynaud, Bernard, Huang, Jun, Lokeswarappa, Kiran Garaga, Gedalius, Joel
• Primary Examiner(s): Hang, Vu B
• Application Number:
  US15/385,493
• Time in Patent Office: 896 Days
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 30/0251 Targeted advertisements
• CPC Class Codes: G06Q 30/0276 Advertisement creation
• CPC Class Codes: G06Q 30/0277 Online advertisement
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals
• CPC Class Codes: H04L 67/306 User profiles

Abstract

A system and method are provided for adding user characterization information to a user profile by analyzing user'"'"'s speech. User properties such as age, gender, accent, and English proficiency may be inferred by extracting and deriving features from user speech, without the user having to configure such information manually. A feature extraction module that receives audio signals as input extracts acoustic, phonetic, textual, linguistic, and semantic features. The module may be a system component independent of any particular vertical application or may be embedded in an application that accepts voice input and performs natural language understanding. A profile generation module receives the features extracted by the feature extraction module and uses classifiers to determine user property values based on the extracted and derived features and store these values in a user profile. The resulting profile variables may be globally available to other applications.

Claims

1. A non-transitory computer-readable medium storing instructions, which when executed by a processor of a car navigation system, cause the processor to:

extract an acoustic feature from user speech;

derive a second acoustic feature from the extracted acoustic feature;

use an accent classifier on the derived second acoustic feature to determine a value of an accent property;

assign the value to the accent property in a user profile; and

generate, by the car navigation system, voice instruction addressed to the user, the voice instruction bearing an accent that corresponds to the value of the accent property.

2. The non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by a processor of a car navigation system, cause the car navigation system to:

extract a linguistic feature from user speech;

determine a value of a region property in the user profile based on the extracted linguistic feature; and

generate, by the car navigation system, voice instructions addressed to the user, the voice instructions comprising words from a vocabulary that corresponds to the value of the region property.

3. The non-transitory computer-readable medium of claim 1, wherein the classifier is trained using machine learning.

4. The non-transitory computer-readable medium of claim 1, wherein the classifier uses pitch statistics.

5. The non-transitory computer-readable medium of claim 1, wherein the extracted acoustic feature comprises at least one of:

a cepstrum and a spectrogram.

6. The non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by a processor of a car navigation system, cause the car navigation system to:

compare the value of the accent property to location context information to determine compatibility,

wherein the assigning of the value to the accent property in the user profile is performed responsive to determining that the value of the accent property is compatible with the location context information.

7. A method of advertisement selection in an online ad system, the method comprising:

extracting an acoustic feature from user speech;

deriving a second acoustic feature from the extracted acoustic feature;

using an accent classifier on the derived second acoustic feature to determine a value of an accent property;

assigning the value to the accent property in a user profile; and

selecting, by the online ad system, an ad for display, the ad being aimed at users of a specific demographic group indicated by the value of the accent property.

8. The method of claim 7, further comprising:

extracting a linguistic feature from user speech;

determining a value of a region property in the user profile based on the extracted linguistic feature; and

selecting, by the online ad system, an ad for display, the ad comprising words from a vocabulary that corresponds to the value of the region property.

9. The method of claim 7, wherein the classifier is trained using machine learning.

10. The method of claim 7, wherein the classifier uses pitch statistics.

11. The method of claim 7, wherein the extracted acoustic feature comprises at least one of:

a cepstrum and a spectrogram.

12. The method of claim 7, further comprising:

comparing the value of the accent property to location context information to determine compatibility,

wherein the assigning of the value to the accent property in the user profile is responsive to determining that the value of the accent property is compatible with the location context information.

13. The method of claim 7, wherein the ad is a music ad.

14. A method of advertisement selection in an online ad system, the method comprising:

extracting an acoustic feature from user speech;

deriving a second acoustic feature from the extracted acoustic feature;

using an age classifier on the derived second acoustic feature to determine a value of an age range property;

assigning the value to the age range property in a user profile; and

selecting, by the online ad system, an ad for display, the ad being aimed at users having an age within the age range assigned to the age range property.

15. The method of claim 14, further comprising:

extracting a linguistic feature from user speech;

determining the value of the age range property in the user profile based on the extracted linguistic feature; and

selecting, by the online ad system, an ad for display, the ad comprising words from a vocabulary that corresponds to the value of the age range property.

16. The method of claim 14, wherein the classifier is trained using machine learning.

17. The method of claim 14, wherein the classifier uses pitch statistics.

18. The method of claim 14, wherein the extracted acoustic feature comprises at least one of:

a cepstrum and a spectrogram.

19. The method of claim 14, wherein the ad is a music ad.

20. A non-transitory computer-readable medium storing instructions, which when executed by a processor of an online add system, cause the processor to:

extract an acoustic feature from user speech;

derive a second acoustic feature from the extracted acoustic feature;

use an age classifier on the derived second acoustic feature to determine a value of an age range property;

assign the value to the age range property in a user profile; and

select, by the online ad system, an ad for display, the ad being aimed at users having an age within the age range assigned to the age range property.

21. The car navigation system of claim 20, further comprising instructions that, when executed by the one or more processors, cause the car navigation system implement actions comprising:

extracting a linguistic feature from user speech;

determining a value of a region property in the user profile based on the extracted linguistic feature; and

generating, by the car navigation system, voice instructions addressed to the user, the voice instructions comprising words from a vocabulary that corresponds to the value of the region property.

22. The car navigation system of claim 20, wherein the classifier is trained using machine learning.

23. The car navigation system of claim 20, wherein the classifier uses pitch statistics.

24. The car navigation system of claim 20, wherein the extracted acoustic feature comprises at least one of:

a cepstrum and a spectrogram.

25. The car navigation system of claim 20, further comprising instructions that, when executed by the one or more processors, cause the car navigation system implement actions comprising:

comparing the value of the accent property to location context information to determine compatibility, wherein the assigning of the value to the accent property in the user profile is performed responsive to determining that the value of the accent property is compatible with the location context information.

26. A non-transitory computer-readable medium storing instructions for advertisement selection, which when executed by a processor of an online ad system, cause the processor to implement a method comprising:

extracting an acoustic feature from user speech;

deriving a second acoustic feature from the extracted acoustic feature;

using an accent classifier on the derived second acoustic feature to determine a value of an accent property;

assigning the value to the accent property in a user profile; and

selecting, by the online ad system, an ad for display, the ad being aimed at users of a specific demographic group indicated by the value of the accent property.

27. The non-transitory computer-readable medium of claim 26, wherein the method implemented by the processor further comprises:

extracting a linguistic feature from user speech;

determining a value of a region property in the user profile based on the extracted linguistic feature; and

selecting, by the online ad system, an ad for display, the ad comprising words from a vocabulary that corresponds to the value of the region property.

28. The non-transitory computer-readable medium of claim 26, wherein the classifier is trained using machine learning.

29. The non-transitory computer-readable medium of claim 26, wherein the classifier uses pitch statistics.

30. The non-transitory computer-readable medium of claim 26, wherein the extracted acoustic feature comprises at least one of:

a cepstrum and a spectrogram.

31. The non-transitory computer-readable medium of claim 26, the method further comprising:

comparing the value of the accent property to location context information to determine compatibility, wherein the assigning of the value to the accent property in the user profile is responsive to determining that the value of the accent property is compatible with the location context information.

32. The non-transitory computer-readable medium of claim 26, wherein the ad is a music ad.