Methods and vehicles for processing voice input and use of tone/mood in voice input to select vehicle response

• US 9,672,823 B2
• Filed: 11/14/2016
• Issued: 06/06/2017
• Est. Priority Date: 04/22/2011
• Status: Active Grant

• Current Assignee: Emerging Automotive, LLC
• Original Assignee: Emerging Automotive, LLC
• Inventors: Penilla, Angel A., Penilla, Albert S.
• Primary Examiner(s): Fantu, Yalkew
• Application Number:
  US15/351,422
• Publication Number:
  US 20170061965A1
• Time in Patent Office: 204 Days
• Field of Search: 320109
• CPC Class Codes: B60R 16/0373 Voice control in general G10L
• CPC Class Codes: G01C 21/3641 Personalized guidance, e.g. limited guidance on previously travelled routes
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 17/04 Training, enrolment or model building
• CPC Class Codes: G10L 17/06 Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G10L 25/45 characterised by the type of analysis window
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: G10L 25/63 : for estimating an emotional state
• CPC Class Codes: G10L 25/84 : for discriminating voice from noise
• CPC Class Codes: G10L 25/90 : Pitch determination of speech signals
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/306 : User profiles

Abstract

Methods, systems and cloud processing are provided for coordinating and processing user input provided to vehicles during use. One example is for processing voice inputs at a vehicle to identify a mood of a user and then modifying or customizing the vehicle response based on the detected mood, physical characteristic and/or physiological characteristic of the user. One example includes a vehicle having an on-board computer for processing voice input. The vehicle having a microphone interfaced with the on-board computer and memory for storing a sample of audio data received from the microphone. The audio data is a voice input directed to the vehicle. A processor of the on-board computer is configured to process the sample of audio data to identify markers in frequency and/or magnitude. The markers are used to define an audio signature for the voice input, and the audio signature is used to identify a voice profile. The voice profile is used to identify a vehicle response for the voice input, and the voice profile is associated with tone of voice used in the voice input. The vehicle response acts to direct a vehicle system function to take an action based on the voice input and the vehicle response is adjusted based on the tone of voice. The tones of voice are associated to inferred moods of the user which include one or more of a normal mood, a frustrated mood, an agitated mood, an upset mood, a hurried mood, an urgency mood, a rushed mood, a stressed mood, a calm mood, a passive mood, a sleepy mood, a happy mood, or an excited mood, or combinations of two or more thereof. The action to be taken is based on the voice input and is one of a command to input a setting of the vehicle, a command requesting information, a request to access data, a request to communicate, or a combination thereof.

Claims

1. A method for processing voice inputs at a vehicle, comprising,

sending, to a server, data from the vehicle, the vehicle including an on-board computer for processing instructions for the vehicle and processing wireless communication to exchange data with the server;

receiving, at the vehicle, data for a user account to use the vehicle, the data used to identify a user profile of a user;

receiving, from the server, data for identifying voice profiles for the user profile, each voice profile being associated with a tone identifier, the voice profiles for the user being learned from a plurality of voice inputs made to the vehicle or other vehicles by the user, while using the user account, in one or more prior sessions of use;

receiving, by the on-board computer, a voice input, and for the voice input,

processing, by on-board computer, the voice input to identify a voice profile that is correlated to the voice input; and

generating a vehicle response for the voice input, the vehicle response is moderated in that the vehicle response is based in part on the tone identifier of the identified voice profile;

wherein the vehicle response is one among a plurality of alternate vehicle response for the voice input as defined by current predefined mappings between the tone identifier and the plurality of alternate vehicle responses.

2. The method of claim 1, wherein the tone identifiers are used to infer a mood of the user and/or a dialect of the user.

3. The method of claim 2, wherein the mood of the user includes one or more of a normal mood, a frustrated mood, an agitated mood, an upset mood, a hurried mood, an urgency mood, a rushed mood, a stressed mood, a calm mood, a passive mood, a sleepy mood, a happy mood, or an excited mood, or combinations of two or more thereof.

4. The method of claim 2, wherein the dialect of the user includes one or more of a language dialect, a country dialect, a geographic region dialect, an age dialect, or combinations of two or more thereof.

5. The method of claim 1, wherein the user account is accessed using login data at the vehicle, the login data defined by credentials of the user, wherein updates to the voice profiles and tone identifiers is processed periodically or upon login by the user profile at the vehicle.

6. The method of claim 5, wherein the login data includes one or more of a password, a wireless login code, a biometric identifier, a gesture, or a pairing between a device of the user and logic of the on-board computer.

7. The method of claim 1, wherein the data received from the server for identifying voice profiles for the user profile include either identifying information to enable selection of voice profiles currently stored in storage of the vehicle or storage of a user device present in the vehicle, or the data received from the server includes one or more voice profiles and/or tone identifiers stored on the server in association with the user account for the user.

8. The method of claim 1, wherein the server stores a copy of one or more of the voice profiles and associated tone identifiers, and storage of the vehicle stores a copy of one more or more of the voice profiles and associated tone identifiers, wherein from time to time, copies stored by the server and stored by the vehicle are synchronized.

9. A method for processing voice input for a vehicle, comprising,

receiving voice input of a user at a microphone of the vehicle to command input or request information from a voice input interface of the vehicle, the voice input interface being in communication with the microphone, the voice input interface providing communication with an application executed by a processor, the processor being of electronics of the vehicle and/or of a portable device paired with the vehicle;

analyzing the voice input of the user to identify a voice profile for the voice input;

selecting a vehicle response from among a plurality of vehicle responses for the voice profile, the selection of the vehicle response uses a mapping of tone identifiers to select one of the plurality of vehicle responses,

the mapping being customizable for the user, and the mapping being stored in association with one or more of a user account on storage of the vehicle, on storage of the portable device, and on storage associated with a server, the server being managed by a cloud processing system that manages a plurality of user accounts for a plurality of users of vehicles configured with wireless communication circuitry for communicating with the cloud processing system;

receiving by electronics of the vehicle updates to settings including updates to the mapping based on learned behavior of the user when processing the one or more voice inputs over time; and

wherein the vehicle response is configured to make or cause one of an operational setting to the vehicle, command the input, request information, provide an input to an application, or cause retrieval of data from a data source responsive to the voice input.

10. The method of claim 9, wherein each tone identifiers is related to a tone detected in the voice input of the user.

11. The method of claim 9, wherein receiving the voice input further includes,

capturing an audio sample of the voice input;

processing the audio sample to remove ambient noise;

identifying frequency and/or magnitude characteristics in the audio sample

generating signature markers for the audio sample based on the frequency and/or magnitude characteristics, and using the signature markers to identify the voice profile for the voice input.

12. The method of claim 11, wherein the analyzing of the voice input to identify voice profiles includes,

filtering the voice profiles associated with the user account of the user to find a voice profile having a threshold level of signature markers to the audio sample associated with the voice input.

13. The method of claim 11, wherein the tone identifiers infer a mood of the user, and the mood of the user includes one or more of a normal mood, a frustrated mood, an agitated mood, an upset mood, a hurried mood, an urgency mood, a rushed mood, a stressed mood, a calm mood, a passive mood, a sleepy mood, a happy mood, an excited mood, or combinations of two or more thereof.

14. The method of claim 9, wherein the voice input includes a tone that is used to identify the tone identifiers and each tone identifier is configurable to set a level of information to be provided back to the user when providing the voice input.

15. The method of claim 14, wherein the level of information is configured to additionally include advertising data, or discount data, or related information that is in addition to the requested information or instructed command found in the voice input.

16. The method of claim 14, wherein setting the level of information to be provided assists in moderating when to provide additional information to the user or when to avoid providing additional information to the user.

17. A vehicle including an on-board computer for processing voice input, comprising,

a microphone interfaced with the on-board computer;

memory for storing a sample of audio data received from the microphone, the audio data being a voice input directed to the vehicle; and

a processor of the on-board computer configured process the sample of audio data to identify markers in frequency and/or magnitude, the markers used to define an audio signature for the voice input of a user, the audio signature being used to identify a voice profile, the voice profile used to identify a vehicle response for the voice input, the voice profile is associated with tone of voice used in the voice input,

wherein the vehicle response acts to direct a vehicle system function to take an action based on the voice input;

wherein the vehicle response is adjusted based on the tone of voice, the tones of voice are associated to inferred moods of the user which include one or more of a normal mood, a frustrated mood, an agitated mood, an upset mood, a hurried mood, an urgency mood, a rushed mood, a stressed mood, a calm mood, a passive mood, a sleepy mood, a happy mood, or an excited mood, or combinations of two or more thereof;

wherein the action to be taken based on the voice input is one of a command to input a setting of the vehicle, a command requesting information, a request to access data, a request to communicate, or a combination thereof.

18. The vehicle of claim 17, wherein the voice profile is one of a plurality of voice profiles of a user associated with predefined tones of voice, and each voice profile is mapped to select vehicle responses.

19. The vehicle of claim 18, wherein the mapping being customizable for the user, and the mapping being stored in association with one or more of a user account on storage of the vehicle, on storage of a portable device that is associated with the vehicle, and on storage associated with a server that is connectable to the vehicle, the server being managed by a cloud processing system that manages a plurality of user accounts for a plurality of users of vehicles configured with wireless communication circuitry for communicating with the cloud processing system.

20. The vehicle of claim 19, wherein the mapping is customizable based on learned behavior of the user;

the learned behavior is inferred from inputs and/or settings made to a vehicle when the user account of the user was active on one or more vehicles, the cloud processing system storing inputs and/or settings made to a vehicle in a database for processing said inferring of the learned behavior, said processing is configured to use machine learning.