Text-based searching of image data

• US 8,533,204 B2
• Filed: 09/02/2011
• Issued: 09/10/2013
• Est. Priority Date: 09/02/2011
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Perronnin, Florent, Saunders, Craig John, Serrano, Jose Antonio Rodriquez
• Primary Examiner(s): Vy, Hung T
• Application Number:
  US13/224,373
• Publication Number:
  US 20130060786A1
• Time in Patent Office: 739 Days
• Field of Search: 707/705, 707/748, 707/758, 707/999.104, 707/749, 707/999.107, 382/128, 382/279, 382/118
• US Class Current: 707/749
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06V 20/625 License plates
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/18152 Extracting features based on a plurality of salient regional features, e.g. "bag of words"
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

A method and system are disclosed for conducting text-based searches of images using a visual signature associated with each image. A measure of string similarity between a query and an annotation associated with each entry in a first database is computed, and based upon the computed string similarity measures, a set of entries from the first database is selected. Each entry of the first database also includes an associated visual signature. At least one entry is then retrieved from a second database based upon a measure of visual similarity between a visual signature of each of the entries in the second database and the visual signatures of the entries in the selected set. Information corresponding to the retrieved entries from the second database is then generated.

Claims

1. A method for text-based searching of image data comprising:

with a computer processor, computing a measure of string similarity between a query and an annotation associated with each of a plurality of entries in a first database, the string similarity measure corresponding to a number of edits to at least one of the query or annotation;

based on the computed string similarity measures, selecting a set of the entries from the associated first database, a visual signature being associated with each entry;

retrieving at least one entry from a second database, based on a measure of visual similarity between a visual signature of each of a plurality of entries in the second database and the visual signatures of the entries in the selected set; and

generating information corresponding to the least one of the retrieved entries.

2. The method of claim 1, wherein the retrieving entries comprises comparing a visual signature of each of the plurality of entries in the second database to the visual signature of each of the selected set so as to determine a similarity therebetween.

3. The method of claim 2, wherein the retrieving comprises ranking entries in the second database based on the similarity of the visual signatures and selecting a subset of ranked entries from the second database, and wherein the information corresponds to the selected subset of ranked entries.

4. The method of claim 1, wherein the retrieving entries comprises:

calculating a combined similarity for each of the plurality of entries in the second database based on the computed measures of visual similarity; and

retrieving entries from the second database based on the combined similarity associated therewith.

5. The method of claim 4, wherein calculating a combined similarity comprises combining, for each entry in the second database, each calculated similarity between the entry in the second database and the selected set.

6. The method of claim 4, wherein calculating a combined similarity comprises computing a combination function that is a weighted average of the individual similarities.

7. The method of claim 6, further comprising calculating a weight during calculation of the combined similarity in accordance with a distance between the query and an annotation of each entry in the associated first database.

8. The method of claim 1, wherein each visual signature is a representation of an image corresponding to a respective entry in one of the first and second databases.

9. The method of claim 8, wherein each visual signature is a vectorial representation describing a distribution of local features extracted from patches of the image.

10. The method of claim 1, further comprising:

calculating a combined signature corresponding to an optionally weighted average of each visual signature of the selected set of entries from the first database; and

wherein the computing of the measure of similarity between a visual signature of each of a plurality of entries in a second database and the visual signatures of the entries in the selected set comprises combining, for each entry in the second database, the combined signature of the selected set of entries and the entry in the second database.

11. The method of claim 1, wherein the query comprises a character string, each character in the string being drawn from a finite set of characters and the annotations comprise character strings, each character in the string being drawn from the same finite set of characters.

12. The method of claim 1, wherein the first and second databases comprise license plate data, and wherein the query comprises at least one of a license plate number and a state.

13. The method of claim 1, wherein the retrieving at least one entry from a second database comprises training a classifier with the visual signatures of the selected set of entries from the first database and applying the trained classifier to the visual signatures of the entries in the second database to identify a subset of entries in the second database.

14. The method of claim 1, wherein the selecting of the set of the entries from the first database comprises selecting entries each having a string distance between the query and the annotation associated with the entry which is below a predetermined threshold.

15. A system for text-based image searching comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory for executing the instructions.

16. A computer program product comprising a non-transitory recording medium storing instructions, which when executed on a computer causes the computer to perform the method of claim 1.

17. A text-based image searching system, comprising:

a processor with access to associated memory, the associated memory storing;

a first associated database comprising a plurality of annotated entries, each entry having a visual signature and annotation associated therewith, and

a second associated database comprising a plurality of entries, each entry having a visual signature associated therewith;

memory in communication with the processor, which stores instructions which are executed by the processor for;

receiving a query which comprises a character string,

computing a string similarity measure between the query character string and an annotation associated with each of a plurality of entries in the first database, the string similarity measure corresponding to a number of edits to transform at least one of the query or annotation into the other,

based on the computed string similarity measures, selecting a set of the entries from the associated first database, a visual signature being associated with each entry,

retrieving at least one entry from a second database, based on a computed visual similarity measure which is based on visual signatures of each of a plurality of entries in the second database and the visual signatures of the entries in the selected set from the associated first database, and

outputting information corresponding to the least one of the retrieved entries.

18. The system of claim 17, further comprising a classifier which is trained on the visual signatures of the plurality of entries to retrieve entries in the second database based on their visual signatures.

19. The system of claim 17, wherein the associated first and second databases comprise at least one of license plate data and driving license data.

20. A method for text-based searching of image data comprising:

receiving a query into memory, the query comprising a character string, each of the characters in the string being selected form a finite set of characters;

with a computer processor, calculating a string similarity measure between the character string of the query and an annotation associated with each of a plurality of visual signatures of captured images stored in a first database, the annotation comprising a character string, each of the characters in the string being selected from the finite set of characters;

selecting a first set of visual signatures from the plurality of visual signatures in the first database based on the computed string similarity measures, the computed string similarity measures corresponding to a number of edits to at least one of the query or annotation;

retrieving a set of images from a second database based on a computed measure of similarity between the visual signatures in the first set of visual signatures and visual signatures of images in the second database.

21. The method of claim 20, further comprising, generating a representation of the retrieved images for display to a user on an associated user interface.

22. The method of claim 20, further comprising calculating a weight in accordance with a distance between the query and an annotation of each entry in the first database, wherein the retrieving the set of images from the second database is also based on the computed weights.

23. The method of claim 20, wherein the first and second databases comprise at least one of license plate images and driving license images.

24. The method of claim 20, wherein the query corresponds to a text string present in an image.