Image search using text-based elements within the contents of images

• US 9,323,784 B2
• Filed: 12/09/2010
• Issued: 04/26/2016
• Est. Priority Date: 12/09/2009
• Status: Active Grant

• Current Assignee: Kyocera Corporation
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: King, Martin T., Mannby, Claes-Fredrik, Smith, Michael J.
• Primary Examiner(s): Rostami, Mohammad S
• Application Number:
  US12/964,662
• Publication Number:
  US 20110153653A1
• Time in Patent Office: 1,965 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5846 using extracted text

Abstract

A mobile device searches for electronic content. The mobile device captures an image from a rendered document, and searches for an electronic version of the image using characteristics of the image and using text within the contents of the image. The mobile device receives a result for the search based upon the image characteristics and the text within the context of the image.

Claims

1. A method for searching for electronic content, the method comprising:

receiving, at a search system, a query image captured from a rendered document;

extracting, from the query image, (i) one or more textual keywords, and (ii) one or more non-text-based, image features;

selecting, from among a set of images, one or more result images that have one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image;

after the one or more result images that have one or more extracted textual keywords have been selected, selecting, from the one or more result images, a subset of the one or more result images that have one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image; and

in response to receiving the query image, providing the subset of the one or more result images that have (i) one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image, and (ii) one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image.

2. The method of claim 1, further comprising:

determining that the captured image has a skew; and

adjusting the captured image to correct the skew.

3. The method of claim 2, wherein adjusting the captured image to correct the skew comprises:

determining a correction to the skew for the text; and

applying the correction to the skew to the captured image.

4. The method of claim 1, wherein extracting one or more non-text-based, image features of the query image comprises determining that the image comprises highly gradual content.

5. The method of claim 4, wherein the selection comprises selecting an algorithm configured for use with highly gradual content.

6. The method of claim 1, wherein extracting one or more non-text based, image features of the query image comprises determining that the image comprises high contrast content.

7. The method of claim 6, wherein the selection comprises selecting an algorithm configured for determining edges and relationships of edges in the high contrast content.

8. The method of claim 1, wherein the one or more non-text-based, image features comprises color, texture, and edge information.

9. The method of claim 1, wherein the selecting is further based on image characteristics of the query image, the characteristics of the query image comprising date and time information for the query image.

10. A system, comprising:

logic circuitry, and

tangible non-transitory memory containing computer-readable instructions that, when processed by the logic circuitry, cause the system to perform functions comprising;

receiving, at a search system, a query image captured from a rendered document;

extracting, from the query image, (i) one or more textual keywords, and (ii) one or more non-text-based, image features;

selecting, from among a set of images, one or more result images that have one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image;

after the one or more result images that have one or more extracted textual keywords have been selected, selecting, from the one or more result images, a subset of the one or more result images that have one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image; and

in response to receiving the query image, providing the subset of the one or more result images that have (i) one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image, and (ii) one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image.

11. The system of claim 10, wherein the functions further comprise:

determining that the captured image has a skew; and

adjusting the captured image to correct the skew.

12. The system of claim 11, wherein adjusting the captured image to correct the skew comprises:

determining a correction to the skew for the text; and

applying the correction to the skew to the captured image.

13. The system of claim 10, wherein extracting one or more non-text-based, image features of the query image comprises highly gradual content.

14. The system of claim 13, wherein the selection comprises selecting an algorithm configured for use with highly gradual content.

15. The system of claim 10, wherein extracting one or more non-text based, image features of the query image comprises determining that the image comprises high contrast content.

16. The system of claim 15, wherein the selection comprises selecting an algorithm configured for determining edges and relationships of edges in the high-contrast content.

17. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving, at a search system, a query image captured from a rendered document;

extracting, from the query image, (i) one or more textual keywords, and (ii) one or more non-text-based, image features;

selecting, from among a set of images, one or more result images that have one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image;

after the one or more result images that have one or more extracted textual keywords have been selected, selecting, from the one or more result images, a subset of the one or more result images that have one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image; and

in response to receiving the query image, providing the subset of the one or more result images that have (i) one or more extracted textual keywords that are associated with one or more of the textual keywords that were extracted from the query image, and (ii) one or more extracted non-text-based, image features that are associated with one or more of the non-text-based, image features that were extracted from the query image.