Client device for interacting with a mixed media reality recognition system
First Claim
1. A method for generating and processing a retrieval request for a visual recognition system, the method comprising:
- receiving an image and generating an image query from the image;
receiving audio data associated with the image, the audio data specifying at least one of a location within a document to be retrieved, at least one type of recognition algorithm for recognizing the image, and an order of the at least one type of recognition algorithm that determines a next algorithm used to recognize the image if a previous algorithm fails;
performing command and data recognition on the audio data to produce audio recognition results for improving image recognition, wherein the audio recognition results include a keyword;
performing retrieval of the document from a database of documents based on the image query and the audio recognition results to produce a retrieval result including a document identification, a portion of the document and an x-y location of the image on the portion of the document, wherein performing retrieval of the document includes performing image recognition on the image based on the image query to produce image recognition results from the database of documents, generating confidence scores associated with the image recognition results, modifying the confidence scores using the keyword and identifying the document based on the modified confidence scores; and
providing the document to a user or performing an action based on the document.
1 Assignment
0 Petitions
Accused Products
Abstract
The mobile device includes a client that has a number of modules, and the MMR Gateway and MMR matching unit are implemented as a server that has a number of modules. The implementation of the MMR system as a client and a server is advantageous because the modules may be distributed among the client and the server in a variety of configurations. The present invention includes a capture module, a preprocessing module, a feature extraction module, a retrieval module, a send message module, an action module, a prediction module, a feedback module, a sending module, an MMR database, a streaming module, an e-mail module, a voice recognition system and an audio database. These modules and systems are operational upon the client or the server.
472 Citations
23 Claims
-
1. A method for generating and processing a retrieval request for a visual recognition system, the method comprising:
-
receiving an image and generating an image query from the image; receiving audio data associated with the image, the audio data specifying at least one of a location within a document to be retrieved, at least one type of recognition algorithm for recognizing the image, and an order of the at least one type of recognition algorithm that determines a next algorithm used to recognize the image if a previous algorithm fails; performing command and data recognition on the audio data to produce audio recognition results for improving image recognition, wherein the audio recognition results include a keyword; performing retrieval of the document from a database of documents based on the image query and the audio recognition results to produce a retrieval result including a document identification, a portion of the document and an x-y location of the image on the portion of the document, wherein performing retrieval of the document includes performing image recognition on the image based on the image query to produce image recognition results from the database of documents, generating confidence scores associated with the image recognition results, modifying the confidence scores using the keyword and identifying the document based on the modified confidence scores; and providing the document to a user or performing an action based on the document. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
-
-
12. A method for generating and processing a retrieval request in a distributed visual recognition system, the method comprising:
-
receiving an image and generating an image query from the image; receiving audio data associated with the image, the audio data specifying at least one of a location within a document to be retrieved, at least one type of recognition algorithm for recognizing the image, and an order of the at least one type of recognition algorithm that determines a next algorithm used to recognize the image if a previous algorithm fails; performing command and data recognition on the audio data to produce audio recognition results for improving image recognition, wherein the audio recognition results include a keyword; performing retrieval of the document from a database of documents based on the image query and the audio recognition results, wherein performing retrieval of the document includes performing image recognition on the image based on the image query to produce image recognition results from the database of documents, generating confidence scores associated with the image recognition results, modifying the confidence scores using the keyword and identifying the document based on the modified confidence scores; and generating and sending a first message including a document identification, a portion identification and an x-y location of the image on the portion of the document. - View Dependent Claims (13, 14, 15, 16)
-
-
17. A method for generating and processing a retrieval request for a visual recognition system, the method comprising:
-
receiving an image and generating an image query from the image; receiving audio data and metadata associated with the image, the audio data specifying at least one of a location within a document to be retrieved, at least one type of recognition algorithm for recognizing the image, and an order of the at least one type of recognition algorithm that determines a next algorithm used to recognize the image if a previous algorithm fails; performing command and data recognition on the audio data to produce audio recognition results for improving image recognition, wherein the audio recognition results include a keyword; performing retrieval of the document from a database of documents based on the image query, the audio recognition results and the metadata to produce a retrieval result including a document identification, a portion of the document and an x-y location of the image on the portion of the document, wherein performing retrieval of the document includes performing image recognition on the image based on the image query to produce image recognition results from the database of documents, generating confidence scores associated with the image recognition results, modifying the confidence scores using the keyword and identifying the document based on the modified confidence scores; and performing an action based on the document. - View Dependent Claims (18, 19, 20, 21)
-
-
22. A system for generating and processing a retrieval request for a visual recognition, the system comprising:
-
a processor; a send message module stored on a memory and executable by the processor, the send message module for receiving an image query and audio data associated with an image, the audio data specifying at least one of a location within a document to be retrieved, at least one type of recognition algorithm for recognizing the image, and an order of the at least one type of recognition algorithm that determines a next algorithm used to recognize the image if a previous algorithm fails; a voice recognition module coupled to the send message module, the voice recognition module for performing command and data recognition on the audio data to produce audio recognition results for improving image recognition, wherein the audio recognition results include a keyword; and a retrieval module coupled to the send message module, the retrieval module for performing retrieval of the document from a database of documents based on the image query and the audio recognition results to produce a retrieval result including a document identification, a portion of the document and an x-y location of the image on the portion of the document, wherein performing retrieval of the document includes performing image recognition on the image based on the image query to produce image recognition results from the database of documents, generating confidence scores associated with the image recognition results, modifying the confidence scores using the keyword and identifying the document based on the modified confidence scores; wherein the send message module provides the document to a user or performs an action based on the document retrieved by the retrieval module.
-
-
23. A non-transitory computer readable storage medium comprising a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:
-
receive an image and generate an image query from the image; receive audio data associated with the image, the audio data specifying at least one of a location within a document to be retrieved, at least one type of recognition algorithm for recognizing the image, and an order of the at least one type of recognition algorithm that determines a next algorithm used to recognize the image if a previous algorithm fails; perform command and data recognition on the audio data to produce audio recognition results for improving image recognition, wherein the audio recognition results include a keyword; perform retrieval of the document from a database of documents based on the image query and the audio recognition results to produce a retrieval result including a document identification, a portion of the document and an x-y location of the image on the portion of the document, wherein performing retrieval of the document includes performing image recognition on the image based on the image query to produce image recognition results from the database of documents, generating confidence scores associated with the image recognition results, modifying the confidence scores using the keyword and identifying the document based on the modified confidence scores; and providing the document to a user or performing an action based on the document.
-
Specification