Method and system for searching for information on a network in response to an image query sent by a user from a mobile communications device

• US 7,949,191 B1
• Filed: 04/04/2007
• Issued: 05/24/2011
• Est. Priority Date: 04/04/2007
• Status: Expired due to Fees

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Ramkumar, Gurumurthy D, Bhattacharyya, Supratik, Manmatha, Raghavan, Bhargava, Gautam, Ruzon, Mark
• Primary Examiner(s): Le; Brian Q
• Assistant Examiner(s): Bayat; Ali
• Application Number:
  US11/732,858
• Time in Patent Office: 1,511 Days
• Field of Search: 382/173, 382/176, 382/190, 382/181, 382/224, 382/209, 382/305, 707/100, 707/3, 707/6, 709/219
• US Class Current: 382/209
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/268 Lexical context

Abstract

Image-based searching for information on a network is provided in response to an image query sent by a user. The image query includes an image captured using a mobile communications device with a camera. The image is processed to detect any text present in the image, and any detected text can be analyzed using a process such as optical character recognition (OCR). The analyzed text is used to search for matches in at least one corresponding domain database, selected from various domain databases present in the network. Thereafter, one or more selected matches and any additional related information can be sent to the user as one or more results for the submitted image query.

Claims

1. A computer-implemented method for searching for information on a network in response to an image query sent by a user, comprising:

under control of one or more computer systems configured with executable instructions,

receiving an image query sent from a mobile communications device, the image query including at least one image taken using a camera of the mobile communications device;

processing the at least one image to detect any text present in the at least one image and to determine geometry information pertaining to the text, the geometry information including at least one of position and geometry information for the text in the at least one image;

automatically recognizing the text from at least one portion of the at least one image corresponding to the geometry information of the image;

determining matches for the detected text in at least one domain database on the network, the at least one domain database selected based at least in part upon the image query by;

removing words in the text that are not present in a pre-identified dictionary;

performing an N-gram match by counting a number of N-grams in common between the text and each field of every entry in the at least one domain database, the at least one domain database being selected from various domain databases;

normalizing a count of the matching N-grams to return a score for each field of every entry in the at least one domain database;

determining weighted combinations of the scores from multiple fields to compute a final score for every entry in the at least one domain database;

ranking entries in the at least one domain database based on the final scores; and

retrieving relevant information for one or more top-ranked entries in the at least one domain database; and

sending one or more of the identified matches to the mobile device for display to the user.

2. The method of claim 1, wherein the network includes a plurality of domain databases, and

wherein the image query further comprises a text query portion along with the image, the text query portion including information corresponding to one of the plurality of domain databases.

3. The method of claim 1, wherein the image is one of a black and white image and a colored image.

4. The method of claim 1, wherein processing the at least one image further comprises transforming the image to improve a quality of the image, and

wherein automatically recognizing the text comprises using at least one optical character recognition process on the at least one portion of the at least one image.

5. The method of claim 4, wherein the quality of the image is improved using an adaptive image normalization algorithm.

6. A computer-implemented method for searching for information on a network in response to an image query sent by a user, comprising:

under control of one or more computer systems configured with executable instructions,

receiving an image query from a user, the image query including at least one image captured by the user;

detecting a presence of text in the at least one image and extracting geometry information corresponding to the text, the geometry information including at least one of a position of the detected text in the at least one image and a size of the detected text;

transforming the image to improve a quality of the image;

recognizing the detected text in at least one portion of the image using an optical character recognition (OCR) process, the at least one portion determined at least in part using the geometry information;

matching the recognized text to a plurality of words in a domain database based at least in part on an N-gram based matching algorithm by;

removing words in the text that are not present in a pre-identified dictionary;

performing an N-gram match by counting a number of N-grams in common between the text and each field of every entry in at least one domain database, the at least one domain database being selected from various domain databases;

normalizing a count of the matching N-grams to return a score for each field of every entry in the at least one domain database;

determining weighted combinations of the scores from multiple fields to compute a final score for every entry in the at least one domain database;

ranking entries in the at least one domain database based on the final scores; and

retrieving relevant information for one or more top-ranked entries in the at least one domain database; and

sending one or more of the words matching the recognized text to the user as identified matches.

7. A system for searching for information on a network in response to an image query sent by a user, the system comprising:

a communication server operable to receive an image query from a user, the image query including at least one image captured by a user and transmitted using a mobile communications device;

an image processing module operable to process the at least one image to determine text geometry information of the image corresponding to text detected in the image, the image processing module further operable to recognize the text in at least one portion of the at least one image determined based at least in part upon the text geometry information; and

a matching engine operable to determine matches for the recognized text on at least one domain database for a network by;

removing words in the text that are not present in a pre-identified dictionary;

performing an N-gram match by counting a number of N-grams in common between the text and each field of every entry in at least one domain database, the at least one domain database being selected from various domain databases;

normalizing a count of the matching N-grams to return a score for each field of every entry in the at least one domain database;

determining weighted combinations of the scores from multiple fields to compute a final score for every entry in the at least one domain database;

ranking entries in the at least one domain database based on the final scores; and

retrieving relevant information for one or more top-ranked entries in the at least one domain database,

wherein the communication server is further operable to send one or more of the determined matches to the user in response to the image query.

8. The system according to claim 7, wherein the image query is received through a medium selected from the group consisting of an e-mail, a File Transfer Protocol (FTP) server, a message, and a Multi Media Service (MMS).

9. The system according to claim 7, wherein the image processing module comprises:

an image processor operable to locate and determine the geometry information in the at least one image; and

an optical character recognition (OCR) module operable to recognize the text from the at least one image.

10. The system according to claim 7, wherein the matching engine comprises a match processor and a score processor, the match processor operable to evaluate N-gram based matches between the detected text and entries in a domain database, the score processor operable to compute a final score for each entry in the domain database based on a commonality of N-gram matches between the recognized text and the entries.

11. The system according to claim 10, wherein the match processor further comprises:

a pre-defined dictionary including a plurality of words having at least one common aspect; and

an N-gram evaluator operable to extract and match N-grams of words in the recognized text.

12. A non-transitory computer-readable storage medium having a computer readable program code embodied therein for searching for information on a network in response to an image query sent by a user, the computer readable program code when executed by at least one processor causing the at least one processor to:

receive an image query sent from a mobile communications device, the image query including at least one image;

process the at least one image to detect any text present in the at least one image and to determine geometry information pertaining to the text, the geometry information including at least one of position and geometry information for the detected text;

automatically recognize the text from at least one portion of the at least one image corresponding to the geometry information of the image;

determine matches for the detected text in at least one domain database on the network, the at least one domain database selected based at least in part upon the image query by;

removing words in the text that are not present in a pre-identified dictionary;

performing an N-gram match by counting a number of N-grams in common between the text and each field of every entry in the at least one domain database, the at least one domain database selected from various domain databases;

normalizing a count of the matching N-grams to return a score for each field of every entry in the at least one domain database;

determining weighted combinations of the scores from multiple fields to compute a final score for every entry in the at least one domain database;

ranking entries in the domain database based on the final scores; and

retrieving relevant information for one or more top-ranked entries in the at least one domain database; and

send one or more of the identified matches to the mobile device for display to the user.

13. The non-transitory computer-readable storage medium of claim 12, wherein the network includes a plurality of domain databases, and

wherein the image query further comprises a text query portion along with the image, the text query portion including information corresponding to one of the plurality of domain databases.

14. The non-transitory computer-readable storage medium of claim 12, wherein the image is a black and white image or a color image.

15. The non-transitory computer-readable storage medium of claim 12, wherein the quality of the image is improved using an adaptive image normalization algorithm.