Methods and systems for searching utilizing acoustical context

• US 10,409,860 B2
• Filed: 03/28/2012
• Issued: 09/10/2019
• Est. Priority Date: 03/28/2011
• Status: Active Grant

• Current Assignee: Diablo Canyon Collective LLC
• Original Assignee: Staton Techiya, LLC
• Inventors: Goldstein, Steven W., Cohen, Jordan
• Primary Examiner(s): Le, Thu Nguyet T
• Application Number:
  US14/007,482
• Publication Number:
  US 20140372401A1
• Time in Patent Office: 2,722 Days
• Field of Search: 707758, 707706
• CPC Class Codes: G06F 16/635 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/638 Presentation of query results
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/9032 Query formulation
• CPC Class Codes: G06F 16/953 Querying, e.g. by the use of web search engines
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10

Abstract

Systems, controllers and methods for contextual-based searching are provided. A system includes one or more devices configured to collect at least one audio stream and a contextual search system. The contextual search system includes a data analyzer and a search engine. The data analyzer is configured to receive the at least one audio stream from among the one or more devices and to determine contextual information from the received at least one audio stream. The search engine is configured to perform a search of at least one search term using the contextual information, to produce a search result.

Claims

1. A system for contextual-based searching comprising:

one or more devices configured to collect at least one audio stream;

one or more sensors for acquiring sensor data related to an environment proximate to one or more of the devices, wherein the one or more sensors are integral or operatively coupled to the one or more devices and include a temperature sensor, a biometric sensor, an image sensor, a light detection sensor, a proximity sensor, a gravity detection sensor, a magnetic field detection sensor, an electrical field detection sensor, a vibration sensor, a pressure senor, a humidity sensor, a moisture sensor, a toxin detection senor, a nutrient detection sensor, an eye-tracker, a pheromone detection sensor, or a combination thereof; and

a contextual search system including;

a data analyzer configured to receive the at least one audio stream and the sensor data from among the one or more devices and to determine contextual information from the received at least one audio stream and the sensor data;

a classifier that transcribes the at least one audio stream and detects an event in the at least one transcribed audio stream;

a search engine configured to perform a search of at least one search term using the contextual information and the detected event to produce a search result,

wherein, in response to the search result, a sensor module is activated and at least one further audio stream is collected by the one or more devices by utilizing the sensor module; and

wherein the contextual search system determines an effectiveness of the search result based on the at least one further audio stream.

2. The system according to claim 1, wherein the system provides the search result to at least one device among the one or more devices.

3. The system according to claim 1, wherein the contextual information indicates a search intent of the search term.

4. The system according to claim 1, wherein the classifier classifies the at least one audio stream into classified information, the data analyzer using the classified information and the at least one search term to determine the contextual information to produce the search result.

5. The system according to claim 1, wherein each audio stream of the at least one audio stream provides a key word, a speech event, a non-speech event, a topic, a transcription, a speaker identity, an emotional state associated with each audio stream, or a combination thereof.

6. The system according to claim 1, wherein the search result is produced from contextual information obtained from at least one or more sensors proximate to each other or from at least one or more sensors mounted in or on separate devices that are proximate to each other.

7. The system according to claim 1, wherein the data analyzer is configured to receive non-acoustic information and wherein the non-acoustic information includes at least one of an image or biometric information.

8. The system according to claim 1, further including a storage device configured to store at least one of the at least one audio stream, non-acoustic information, the contextual information, the at least one search term, the search result or a log of classified information of non-acoustic information over a period of time.

9. The system according to claim 1, wherein each device of the one or more devices has at least a communication capability for interchanging information with at least one of others of the one or more devices or a server.

10. The system according to claim 9, wherein at least one of the server or the one or more of the devices includes the contextual search system.

11. The system according to claim 1, wherein each of at least some of the devices include a user interface, the search being performed responsive to an indication received from the user interface.

12. The system according to claim 1, wherein the contextual search system continuously monitors the at least one audio stream, and a further search is performed automatically responsive to an indication in the monitored audio stream.

13. The system according to claim 1, wherein each of at least some of the one or more devices have the one or more of the sensors for acquiring the sensor data related to the environment proximate to the one or more devices and at least one or more microphones configured to collect the at least one audio stream proximate to the one or more devices.

14. The system according to claim 1, wherein the one or more devices are configured to collect the at least one audio stream using at least one of the electromechanical transducer, an optical transducer, or a piezoelectric transducer.

15. The system according to claim 1, wherein the one or more devices are configured to collect at least two audio streams from transducers including at least one ear canal microphone and an external microphone.

16. The system according to claim 1, wherein at least one of the one or more devices includes at least one of a mobile telephone device, an earpiece, a hearing aid, a watch, a television remote, a car key fob, a portable game controller, a navigation device, or a computer having at least one sensor.

17. The system according to claim 1, wherein further comprising an occlusion element configured to fully or substantially isolate an ear canal from an external audio field.

18. The system according to claim 1, wherein the one or more devices are configured to collect the least one audio stream using least one of a throat microphone, a jaw microphone, or a bone conduction microphone.

19. A controller for contextual-based searching, the controller configured to interact with one or more devices, each device of the one or more devices having at least a communication capability for interchanging information with at least one of others of the one or more devices, the controller or a communication system, at least one of the one or more of the devices or the controller configured to collect at least one audio stream and sensor data proximate to the one or more of the devices or the controller, the controller comprising:

a contextual search system including;

a data analyzer configured to receive the at least one audio stream and the sensor data from among at least one of the one or more devices or the controller and to determine contextual information from the received at least one audio stream and the sensor data,

a classifier that transcribes the at least one audio stream and detects an event in the at least one transcribed audio stream;

a search engine configured to perform a search of at least one search term using the contextual information and the detected event to produce a search result,

wherein, in response to the search result, a sensor module is activated and at least one further audio stream is collected by the one or more devices by utilizing the sensor module, and

wherein the contextual search system determines an effectiveness of the search result based on the at least one further audio stream.

20. The controller according to claim 19, further including a communication module configured to receive the at least one audio stream from among the one or more devices.

21. The controller according to claim 19, wherein two or more transducers from at least one of the one or more of the devices are selected across a network and used as a beamforming array.

22. The controller according to claim 19, wherein each of at least some of the one or more devices have the sensor module including a microphone and at least one of a temperature sensor, a biometric sensor, an image sensor, a light detection sensor, a proximity sensor, a gravity detection sensor, a magnetic field detection sensor, an electrical field detection sensor, a vibration sensor, a pressure sensor, a humidity sensor, a moisture sensor, a toxin detection sensor, a nutrient detection sensor or a pheromone detection sensor.

23. The controller according to claim 19, wherein the controller provides the search result to at least one device among the one or more devices.

24. The controller according to claim 19, wherein the classifier classifies information from the audio streams into classified information, the data analyzer using the classified information and the at least one search term to determine the contextual information.

25. The controller according to claim 24, wherein the data analyzer is configured to receive non-acoustic information associated with the respective device, the data analyzer using at least one of the non-acoustic information or further classified information from the non-acoustic information to determine the contextual information.

26. The controller according to claim 19, further comprising a storage device configured to store a predetermined period of time of speech and text before and after a search trigger initiation.

27. The controller according to claim 19, wherein the contextual search system continuously monitors the at least one audio stream, and a further search is performed automatically responsive to an indication in the monitored audio stream.

28. The controller according to claim 19, further including a user interface, the search being performed responsive to an indication received from the user interface.

29. A method for contextual-based searching comprising:

collecting at least one audio stream from one or more devices;

collecting sensor data from the one or more devices;

determining, by a data analyzer, contextual information from the at least one audio stream and the sensor data received from the one or more devices;

transcribing the at least one audio stream and detecting an event in the at least one transcribed audio stream;

performing, by a search engine, a search of at least one search term using the contextual information and the detected event to produce a search result;

wherein the contextual information is used to disambiguate a search intent of the search term;

activating, in response to the search result, a sensor module;

collecting, in response to the search result, at least one further audio stream from the one or more devices by utilizing the sensor module; and

determining an effectiveness of the search result based on the at least one further audio stream.

30. The method according to claim 29, the method further including providing the search result to at least one device among the one or more devices, the search result being presented at least one of visually, aurally or by a haptic interface.

31. The method according to claim 30, the method further comprising:

determining further contextual information from the at least one further audio stream; and

determining the effectiveness of the search result from the further contextual information.

32. The method according to claim 29, wherein the sensor data comprises sensor data from at least one of a temperature sensor, a biometric sensor, a heart rate sensor, an image sensor, a motion sensor, an accelerometer, a location sensor, a light detection sensor, a proximity sensor, a gravity detection sensor, a magnetic field detection sensor, an electrical field detection sensor, a vibration sensor, a pressure sensor, a humidity sensor, a moisture sensor, a toxin detection sensor, a nutrient detection sensor or a pheromone detection sensor.

33. The method according to claim 29, the method further including:

continuously monitoring at least two audio streams; and

automatically performing a further search responsive to an indication in the monitored audio streams.

34. The method according to claim 29, wherein the search is performed responsive to an indication received from at least one of the one or more devices.

35. The method according to claim 34, wherein a first portion of the at least one audio stream prior to the indication and a second portion of the at least one audio stream after the indication are used to determine the contextual information.

36. The method according to claim 29, the method further comprising:

performing a further search of the at least one search term using the search result to produce a further search result.

37. The method according to claim 29, wherein the determining of the contextual information includes classifying the audio stream into classified information in the form of text, the classified information, the sensor data, and the at least one search term being used to determine the contextual information.

38. The method according to claim 37, wherein the classified information includes at least one of a key word, a speech event, a non-speech event, a topic, a transcription, a speaker identity or an emotional state associated with the audio stream.

39. The method according to claim 37, wherein the determining of the contextual information further includes using at least one of non-acoustic information or further classified information from the non-acoustic information.

40. The method according to claim 39, wherein the non-acoustic information includes at least one of a speaker identity, an image, or biometric information.