Multimodal natural language query system for processing and analyzing voice and proximity-based queries

• US 9,223,776 B2
• Filed: 10/18/2012
• Issued: 12/29/2015
• Est. Priority Date: 03/27/2012
• Status: Active Grant

• Current Assignee: Portal Communications, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Intellectual Group Incorporated
• Inventors: Bernard, David E.
• Primary Examiner(s): Wong, Leslie
• Application Number:
  US13/655,403
• Publication Number:
  US 20130262107A1
• Time in Patent Office: 1,167 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: Y10S 707/918 Location

Abstract

The disclosure provides a natural language query system and method for processing and analyzing multimodally-originated queries, including voice and proximity-based queries. The natural language query system includes a Web-enabled device including a speech input module for receiving a voice-based query in natural language form from a user and a location/proximity module for receiving location/proximity information from a location/proximity device. The query system also includes a speech conversion module for converting the voice-based query in natural language form to text in natural language form and a natural language processing module for converting the text in natural language form to text in searchable form. The query system further includes a semantic engine module for converting the text in searchable form to a formal database query and a database-look-up module for using the formal database query to obtain a result related to the voice-based query in natural language form from a database.

Claims

1. A query system, comprising:

a computing device communicatively coupled to a network and configured to receive a plurality of inputs;

wherein at least one of said inputs is an audio query and another of said inputs is a visual input;

a speech server communicatively coupled to the computing device via the network, wherein the speech server is configured to receive the audio query from the computing device, convert the audio query into a text query and return the text query to said computing device;

a mediated reality server communicatively coupled to the computing device via the network, wherein the mediated reality server is configured to receive and process the visual input to determine data relevant to said visual input and return said data relevant to said visual input to said computing device; and

a remote server communicatively coupled to the computing device via the network, wherein the remote server is configured to receive said text query and at least a portion of said data relevant to said visual input and based on the received text query and the at least a portion of said data relevant to said visual input determine a response to the query.

2. The query system according to claim 1, wherein said speech server, said mediate reality server and said remote server are the same server.

3. The query system according to claim 1, wherein said remote server is configured to communicate said response to said computing device and said computing device is configured to communicate said response to a user.

4. The query system according to claim 3, wherein said computing device is configured to convert said response into audio and play said audio through a speaker.

5. The query system according to claim 3 wherein said computing device is configured to convert said response into a multi-media-based response.

6. The query system according to claim 1 wherein said speech server is further configured to convert said text query from a first language to another language.

7. The query system according to claim 1 wherein said remote server is communicatively coupled to a plurality of databases and wherein said remote server is configured to concurrently search said plurality of databases for a response to said query and rank a response determined from each of said searched databases.

8. The query system according to claim 6, wherein said rank is based on an accuracy of the results.

9. The query system according to claim 1 wherein said remote server is configured to store said query and response for further analysis.

10. A mobile device query method, comprising:

receiving an audio query from a user and a visual input from a visual input device;

transmitting said visual input to a mediated reality server;

said mediated reality server processing said visual input to derive data relevant to said visual input and return said data relevant to said visual input to said mobile device;

transmitting the audio query and the data relevant to said visual input to a remote server; and

receiving a plurality of responses to the audio query from the server, said responses being based at least in part on said data relevant to said visual input, and each of the plurality of responses being ranked by the server using an accuracy algorithm.

11. The mobile device query method according to claim 10 further comprising storing said query and said plurality of responses and using said stored responses to refine responses to an additional audio query.

12. The mobile device query method according to claim 10 further comprising receiving a plurality of queries, said remote server storing said plurality of queries and performing a frequency analysis on said plurality of queries.

13. The mobile device query method according to claim 10 further comprising said remote server receiving a plurality of queries from the user, and storing said plurality of queries and an order in which said plurality of queries is received.