Audio based system and method for in-vehicle context classification

• US 9,311,930 B2
• Filed: 01/28/2014
• Issued: 04/12/2016
• Est. Priority Date: 01/28/2014
• Status: Active Grant

• Current Assignee: Qualcomm technologies international, ltd. (Qualcomm, Inc.)
• Original Assignee: Qualcomm technologies international, ltd. (Qualcomm, Inc.)
• Inventors: Srinivasan, Ramji, Rea, Derrick, Trainor, David
• Primary Examiner(s): Paul, Disler
• Application Number:
  US14/165,902
• Publication Number:
  US 20150215716A1
• Time in Patent Office: 805 Days
• Field of Search: 381/86, 381 58- 59, 381/302, 381/89, 704/226, 704/233
• US Class Current: 1/1
• CPC Class Codes: G10L 25/27 characterised by the analysis technique
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/72 for transmitting results of analysis
• CPC Class Codes: H04R 2499/13 Acoustic transducers and sound field adaptation in vehicles

Abstract

A method of determining contexts for a vehicle, each context corresponding to one or more events associated with the vehicle, for example that the radio is on and a window is open. The method comprises detecting sound activities in an audio signal captured in the vehicle, and assigning context to the vehicle based on the detected sound activities. Non-audio data such as the operational status of a vehicle system or device is used to help assign contexts.

Claims

1. A method of determining contexts and associated sound activities for a vehicle, the method including:

associating a plurality of vehicle contexts with a respective one or more of a plurality of sound activities;

detecting an audio signal in the vehicle;

detecting, based on said detected audio signal, at least one of said sound activities in said audio signal;

subsequent to detecting at least one of said sound activities, identifying a vehicle context associated with said detected at least one of said sound activities;

providing a respective n-gram model for each of said vehicle contexts; and

assigning to said vehicle said at least one identified vehicle context and said detected at least one of said sound activities with which said identified vehicle context is associated, wherein said assigning involves using a history of at least one previously assigned vehicle context together with said n-gram models in identifying said at least one of said vehicle contexts.

2. The method of claim 1, wherein said assigning involves using non-audio vehicle data in determining said at least one of said vehicle contexts.

3. The method of claim 2, wherein said non-audio vehicle data comprises data indicating the operational status of one or more of the vehicle'"'"'s systems or vehicle'"'"'s devices.

4. The method of claim 2, including obtaining said non-audio data from at least one vehicle sensor.

5. The method of claim 4, wherein said at least one sensor is configured to detect the status of any one or more aspects of the vehicle, including a windshield wiper, direction indicator, media player, navigation system, window, sun roof, rain sensor, fan, air conditioning system or telephone system.

6. The method of claim 2, including obtaining said non-audio data from a vehicle control system.

7. The method of claim 6, including obtaining said non-audio data from a control unit of the vehicle.

8. The method of claim 2, wherein said using non-audio vehicle data in determining said at least one of said vehicle contexts involves using said non-audio vehicle data to determine compatibility of at least some of said vehicle contexts with said detected audio signal.

9. The method of claim 1, including detecting said audio signal using at least one microphone.

10. The method of claim 9, wherein said microphone is incorporated into said vehicle such that said audio signal corresponds to sounds in a cabin of the vehicle detected by said at least one microphone.

11. The method of claim 1, including segmenting said audio signal into audio segments, wherein said detecting said at least one of said sound activities involves detecting a respective at least one of said sound activities in each audio segment;

and said assigning involves assigning said a respective at least one of said identified vehicle contexts and said at least one of said associated sound activities in respect of each audio segment.

12. The method of claim 11, including using non-audio vehicle data in determining boundaries for the audio segments during the segmentation process.

13. The method of claim 11, including performing feature extraction on said audio segments to provide a respective frequency-based definition of each audio segment.

14. The method of claim 13, including assuming that temporal sparsity exists in said respective frequency-based definition.

15. The method of claim 11, including organizing said audio segments into respective frames, wherein each frame corresponds to a single sound activity or sound activity model.

16. The method of claim 11, wherein said assigning involves using non-audio vehicle data to determine compatibility of at least some of said vehicle contexts with each audio segment, and wherein said detecting a respective at least one of said sound activities in each audio segment involves detecting sound activities that are associated with said vehicle contexts that are determined to be compatible with said detected audio segment.

17. The method of claim 11, wherein said assigning involves using non-audio vehicle data to determine compatibility of at least some of said vehicle contexts with each audio segment, and wherein said assigning said a respective at least one of said vehicle contexts in respect of each audio segment involves assigning sound activities that are associated with said vehicle contexts that are determined to be compatible with said detected audio segment.

18. The method of claim 11, including providing a plurality of sound activity models, wherein each of said plurality of sound activity models comprises a mathematical representation of a respective one or more of said sound activities, and wherein said detecting at least one of said sound activities in said audio signal involves comparing said audio segments against at least some of said sound activity models.

19. The method of claim 1, including providing a plurality of sound activity models, wherein each of said plurality of sound activity models comprises a mathematical representation of a respective one or more of said sound activities, and wherein said detecting at least one of said sound activities in said detected audio signal involves comparing said audio signal against at least some of said plurality of sound activity models.

20. The method of claim 19, wherein said associating a plurality of vehicle contexts with a respective one or more of said sound activities involves associating each said plurality of vehicle contexts with a respective one or more of said sound activity models corresponding to said respective one or more of said sound activities.

21. The method of claim 19, wherein said assigning involves using non-audio vehicle data to determine compatibility of at least some of said vehicle contexts with said detected audio signal, and wherein said comparing said detected audio signal against at least some of said plurality of sound activity models involves comparing sound activity models that are associated with said vehicle contexts that are determined to be compatible with said detected audio signal.

22. The method of claim 19, wherein said comparing said detected audio signal against at least some of said plurality of sound activity models involves computing a respective matching score for at least some of said sound activity models, comparing said matching scores, and wherein said detecting at least one of said sound activities in said audio signal involves determining which of said sound activities is detected based on said comparison of said matching scores.

23. The method of claim 22, including segmenting said detected audio signal into audio segments, wherein said detecting said at least one of said sound activities involves detecting a respective at least one of said sound activities in each audio segment;

and said assigning involves assigning a respective at least one of said vehicle contexts in respect of each audio segment, and wherein said comparing said audio signal against at least some of said sound activity, said computing a respective matching score for at least some of said plurality of sound activity models, and said determining which of said sound activities is detected are performed in respect of each audio segment.

24. The method of claim 22, wherein said comparing said matching scores involves weighting said matching scores using a respective n-gram model of the respective vehicle context associated with at a respective one of the sound activity models.

25. The method of claim 1, wherein each of said vehicle contexts corresponds to a respective one or more events associated with said vehicle.

26. The method of claim 1, wherein each of said sound activities comprises either a specific sound, or a specific sound type, or a specific sound source, or any combination of one or more sound, one or more sound types and/or one or more sound sources.

27. A system for determining contexts and associated sound activities for a vehicle, the system including:

at least one microphone for detecting an audio signal in the vehicle; and

a context classification system configured to;

associate a plurality of vehicle contexts with a respective one or more of a plurality of sound activities,

detect, based on said detected audio signal, at least one of said sound activities in said audio signal,

subsequent to detecting at least one of said sound activities, identify a vehicle context associated with said detected at least one of said sound activities,

provide a respective n-gram model for each of said vehicle contexts; and

assign to said vehicle said at least one identified vehicle context and said detected at least one of said sound activities with which said identified vehicle context is associated, wherein said assigning involves using a history of at least one previously assigned vehicle context together with said n-gram models in identifying said at least one of said vehicle contexts.

28. A system as claimed in claim 27, wherein said context classification system configured to obtain non-audio data for use in said assignment of vehicle contexts.

29. The system of claim 28, including at least one sensor for detecting non-audio vehicle data and for providing said non-audio data to said context classification system.

30. The system of claim 29, wherein said at least one sensor is configured to detect non-audio vehicle data comprising data indicating the operational status of one or more of the vehicle'"'"'s systems or vehicle'"'"'s devices.

31. The system of claim 28, wherein said context classification system configured to obtain said non-audio data from a vehicle control system.

32. A vehicle audio system comprising a system for determining contexts and associated sound activities for a vehicle, the context determining system including:

at least one microphone for detecting an audio signal in the vehicle; and

a context classification system configured to

associate a plurality of vehicle contexts with a respective one or more of a plurality of sound activities,

detect, based on said detected audio signal, at least one of said sound activities in said audio signal,

subsequent to detecting at least one of said sound activities, identify a vehicle context associated with said detected at least one of said sound activities,

provide a respective n-gram model for each of said vehicle contexts; and

assign to said vehicle said at least one identified vehicle context and said detected at least one of said sound activities with which said identified vehicle context is associated, wherein said assigning involves using a history of at least one previously assigned vehicle context together with said n-gram models in identifying said at least one of said vehicle contexts.

33. The vehicle audio system as claimed in claim 32, including or being co-operable with at least one audio device, wherein the operation of at least one of said at least one audio device is dependent on the assigned at least one of said vehicle contexts and said detected at least one of said sound activities with which said identified vehicle context is associated.

34. The vehicle audio system as claimed in claim 33, wherein said at least one audio device includes, or is co-operable with, any one or more of an audio rendering device, a speech recognition device, a noise cancellation device or a noise enhancement device.

35. The vehicle audio system as claimed in claim 33, wherein said at least one audio device comprises any one or more of a radio, a CD player, a media player, a telephone system, a navigation system or a voice command system.