Media capture and process system

• US 10,276,155 B2
• Filed: 12/22/2016
• Issued: 04/30/2019
• Est. Priority Date: 12/22/2016
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Montantes, James
• Primary Examiner(s): Saint Cyr, Leonard
• Application Number:
  US15/388,683
• Publication Number:
  US 20180182379A1
• Time in Patent Office: 859 Days
• Field of Search: None
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: H03G 5/165 Equalizers; Volume or gain control in limited frequency bands
• CPC Class Codes: H04R 1/326 for microphones H04R1/34 and H04R1/40 take precedence
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2499/11 Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 3/04 for correcting frequency response

Abstract

A user media device may include a microphone array and a communication interface. The microphone array may include an omnidirectional microphone and a directional microphone. The microphone array may be selectively switchable. The communication interface may communicatively couple the user media device with a computer and may transmit audio captured by the microphone array to the computer for transfer to a remote service. The remote service may generate text of the processed audio via natural language processing. The remote service may further perform semantic reasoning of the processed audio via a semantic reasoning engine. The remote service may also generate content based at least in part on the semantic reasoning performed on the processed audio. The curated content may include a report having results of the semantic reasoning organized to demonstrate the results in a meaningful way with respect to the processed audio.

Claims

1. A user media device comprising:

a microphone array configured to capture audio and including an omnidirectional microphone and a directional microphone, the microphone array being selectively switchable with respect to operation of the omnidirectional microphone and the directional microphone such that content of the captured audio is based on a selection setting of the microphone array; and

a communication interface to communicatively couple the user media device with a computer and to transmit audio captured by the microphone array to the computer to transfer to a remote service configured to perform operations comprising;

generating text of the captured audio via natural language processing;

performing semantic reasoning of the generated text via a semantic reasoning engine in which the semantic reasoning includes;

identifying a frequency of use of words included in the text, identifying a context of use of the words in the text, and identifying a subject tag associated with the words;

determining a relevancy of the words based on the semantic reasoning;

accessing user information related to a user of the user media device in which the user information includes;

predefined keywords, a history of the user, and preferences of the user;

generating curated content based on the relevancy of the words included in the text as determined from the semantic reasoning performed on the generated text and based on the user information, wherein the curated content is customized according to the predefined keywords, history of the user, and preferences of the user included in the user information and includes a report having images, external links, analytical data, and statistical data that provide insight about the captured audio; and

]

causing presentation of the curated content on a user interface;

wherein the computer is configured to perform operations comprising;

performing audio processing operations with respect to the captured audio, the audio processing operations including one or more of;

filtering the captured audio and equalizing the captured audio; and

transmitting the processed audio to the user media device, wherein the communication interface of the user media device receives the processed audio and the user media device includes a speaker configured to present the received processed audio.

2. The user media device of claim 1, wherein the microphone array includes a second omnidirectional microphone and a second directional microphone.

3. The user media device of claim 1, wherein audio captured by the omnidirectional microphone is employed to perform noise cancelation relative to the audio captured by the directional microphone.

4. The user media device of claim 1, further comprising a camera.

5. The user media device of claim 1, wherein the user media device is configured to be worn on a head of a user.

6. The user media device of claim 1, wherein the microphone array is selectively switchable based at least in part on control commands received by the user media device from the computer.

7. A method of processing audio captured by a user media device, the method comprising:

capturing audio at a microphone array of the user media device;

transmitting the captured audio from the user media device to a computer communicatively coupled to the user media device;

processing the captured audio at the computer including performing one or more of;

filtering the captured audio and equalizing the captured audio;

transmitting the processed audio from the computer to the user media device, wherein a communication interface of the user media device receives the processed audio and the user media device includes a speaker configured to present the received processed audio;

presenting the processed audio at a speaker of the user media device;

generating text of the captured audio via natural language processing;

performing semantic reasoning of the generated text via a semantic reasoning engine in which the semantic reasoning includes;

identifying a frequency of use of words included in the text, identifying a context of use of the words in the text, and identifying a subject tag associated with the words;

determining a relevancy of the words based on the semantic reasoning;

accessing user information related to a user of the user media device in which the user information includes;

predefined keywords, a history of the user, and preferences of the user;

generating curated content based on the relevancy of the words included in the text as determined from the semantic reasoning performed on the generated text and based on the user information, wherein the curated content is customized according to the predefined keywords, history of the user, and preferences of the user included in the user information and includes a report having images, external links, analytical data, and statistical data that provide insight about the captured audio; and

causing presentation of the curated content on a user interface.

8. The method of claim 7, further comprising:

capturing visual media at a camera of the user media device; and

transmitting the captured visual media from the user media device to the computer.

9. The method of claim 8, further comprising transmitting the captured visual media and one or more tags associated with the captured visual media from the computer to a remote service.

10. The method of claim 7, further comprising:

transmitting the processed audio from the computer to a remote service;

generating, at the remote service, text of the processed audio via natural language processing;

performing semantic reasoning of the generated text via a semantic reasoning engine in which the semantic reasoning includes;

identifying a frequency of use of words included in the text, identifying a context of use of the words in the text, and identifying a subject tag associated with the words;

determining a relevancy of the words based on the semantic reasoning; and

the generating of the curated content is based on the relevancy of the words included in the text as determined from the semantic reasoning performed on the processed audio.

11. A media capture and process system comprising:

a computer;

a user media device including;

a microphone array; and

a communication interface for communicatively coupling the user media device with the computer and for transmitting audio captured by the microphone array to the computer, wherein the computer is configured to perform operations comprising;

performing audio processing operations with respect to the captured audio, the audio processing operations including one or more of;

filtering the captured audio and equalizing the captured audio; and

transmitting the processed audio to the user media device, wherein the communication interface of the user media device receives the processed audio and the user media device includes a speaker configured to present the received processed audio; and

a remote service for receiving the captured audio from the computer and processing the captured audio, the remote service including;

a semantic reasoning engine for performing semantic analysis on the captured audio in which the semantic reasoning includes;

identifying a frequency of use of words included in the captured audio, identifying a context of use of the words in the captured audio, and identifying a subject tag associated with the words and wherein the semantic reasoning engine further determines a relevancy of the words based on the semantic reasoning;

a data curation engine for curating data associated with the captured audio based on the relevancy of the words included in the captured audio as determined from the semantic reasoning to generate curated content that is customized according to predefined keywords, history of the user, and preferences of the user included in user information accessed by the data curation engine and includes analytical data and statistical data that provide insight about the captured audio;

an analytics engine for generating reports based on the captured audio, wherein the report includes images, external links, analytical data, and statistical data that provide insight about the captured audio; and

a network controller that communicates the curated content to a user device such that the curated content is displayed on the user device.

12. The media capture and process system of claim 11, wherein the microphone array includes an omnidirectional microphone and a directional microphone, the microphone array being selectively switchable.

13. The media capture and process system of claim 11, wherein the remote service includes a natural language engine for converting the captured audio to text.

14. The media capture and process system of claim 11, wherein the reports are accessible via an application programming interface (API).

15. The media capture and process system of claim 11, wherein the analytics engine generates the reports based on the captured audio at least in part via statistical analysis of results of the semantic analysis.

16. The media capture and process system of claim 11, wherein the user media device further includes a camera for capturing visual media and the remote service includes a deep learning engine for performing, on the captured visual media, at least one of object detection, context analysis, and image processing.